INFO: Excel May Change the Precision Mode of the Floating-Point Control Word (263213)


The information in this article applies to:

  • Microsoft Office Excel 2003
  • Microsoft Excel 2002
  • Microsoft Excel 2000
  • MSPRESS Microsoft Excel 97 Developer's Kit ISBN 1-57231-498-2
  • Microsoft Excel 97 for Windows
This article was previously published under Q263213

SUMMARY

Microsoft Excel may change the precision mode of the floating-point control word during its execution. Because of this, C/C++ DLLs called from Excel that use the standard run-time library may return slightly different results than expected. The difference is typically less than 10E-15.

MORE INFORMATION

C/C++ DLLs or XLLs designed to be used from Excel should link to the FP10.OBJ library. The Microsoft C/C++ run-time library sets the default internal precision of the math coprocessor (or emulator) to 64 bits. By linking your project with FP10.OBJ, you override this default and set the chip to utilize 80-bit precision. FP10.OBJ must appear before Libc.lib, Libcmt.lib, or Msvcrt.lib on the linker command line. By linking to FP10.OBJ, you should see more consistent results. However, even when utilizing 80-bits, Excel is still limited to 15 digits of precision because of its adherence to the IEEE 754 specification.

REFERENCES

78113 XL: Floating-Point Arithmetic May Give Inaccurate Results

125056 INFO: Precision and Accuracy in Floating-Point Calculations

Visual C++ Programmer's Guide, under "Floating-Point Support".
Modification Type:MajorLast Reviewed:12/12/2003
Keywords:kbFloatPoint kbinfo KB263213
©2003 Microsoft Corporation. All rights reserved.