BUG: FoxPro Math-Precision Problems with SET DECIMALS (114136)
The information in this article applies to:
- Microsoft Visual FoxPro for Windows 3.0
- Microsoft Visual FoxPro for Windows 3.0b
- Microsoft Visual FoxPro for Windows 5.0
- Microsoft Visual FoxPro for Windows 5.0a
- Microsoft Visual FoxPro for Windows 6.0
- Microsoft FoxPro for MS-DOS 2.0
This article was previously published under Q114136 SYMPTOMS
SET DECIMALS TO 2 changes the result when a numeric expression that
evaluates to a decimal value is used in a comparison. For example, type the
following in the Command window:
SET DECIMAL TO 18
? 12/8888888 = 0 && FoxPro returns FALSE
SET DECIMALS TO 2
? 12/8888888 = 0 && FoxPro returns TRUE
Under dBASE IV, both of these expressions will return FALSE.
In FoxPro version 2.6a, a similar decimal-precision problem occurs when any
whole number is used in math problems. For example:
SET DECIMALS TO 18
?12/8888888 && dBASE returns 0.00000135... but FoxPro returns 0
?12/88888888 && dBASE returns 0.0000000000... but FoxPro returns 0
?12.0/8888888 && dBASE returns 0.00000135... but FoxPro returns 0
However the behavior of FoxPro versions 2.5x, 2.6 and Visual FoxPro is
closer to that of dBase. For example:
SET DECIMALS TO 18
?12/8888888 && FoxPro returns 0.000001350000135000
?12/88888888 && FoxPro returns 0
?12.0/8888888 && FoxPro returns 0.000001350000135000
RESOLUTION
If you need greater precision in FoxPro, set the SET DECIMALS command to a
higher value. For greater precision with constants, replace the constants
with memory variables, so that FoxPro performs the calculation at run time.
If you must use constants, express them in the precision that is desired.
For example, use this expression
?12.0000000/88888888 && FoxPro returns 0.00000135...
rather than this one:
?12/88888888 && FoxPro returns 0.0000000000...
STATUS
Microsoft has confirmed this to be a problem in the Microsoft products
listed at the beginning of this article.
MORE INFORMATION
NOTE: FoxPro returns a zero when the first significant digit is in the
seventh position to the right of the decimal; for example, 0.0000001.
| Modification Type: | Major | Last Reviewed: | 5/6/2003 |
|---|
| Keywords: | kbBug KB114136 |
|---|
|