The result of dividing LongLong data by a number is
incorrect
Symptom
The result of dividing LongLong data by a number is incorrect. For
example:
|
1 2 3 |
longLong x, y x = 7129750713734852035 // LongLong-Value with 19 digits y = x / 16 // y gets the value of 445609419608428224, which is wrong, the correct result should be 445609419608428252 |
Not all data of the LongLong type have this problem. If you divide
9223372036854775807 by 16, the result is correct.
Environment
PowerBuilder 2017 R3 or later
Cause
This is a Sybase legacy issue, possibly related to algorithm
accuracy.
Solution
Use the DEC type instead of the LongLong type:
|
1 2 3 |
Dec x, y x = 7129750713734852035 y = x / 16 |
Document get from Powerbuilder help
Thank you for watching.
Subscribe
Login
0 Comments
Oldest