how does arithmetic exactly work in Calc?

[Michel Onoff <michel.onoff@xxxxxx>]

Hi,

I would like to understand how simple arithmetic (+ - * /) works in Calc.
I kind of guess that the underlying internal number representation is
IEEE 754 double (64 bit).
I also guess that when a double is shown on the sheet it is approximated
by a decimal with at most 15 significand digits.

To reproduce the behavior below, use scientific notation with 20 digits
or more.

I have two slightly different number x and x'.
x is 2^-49 (a formula) while x' is 1.77635683940025E-15 (a literal).
Their decimal representation appear equal on the sheet, but they are,
indeed, slightly different internally. You can set them apart as follows:

y = x - 1.7763568394002E-15
y' = x' - 1.7763568394002E-15

that is, by subtracting the same number from x and x'.
y and y' appear differently on the sheet, meaning that x and x' are
different internally to start with.

However, x - x' is exactly 0. In IEEE 754 arithmetic, two numbers are
equal if and only if their difference is 0. That would mean that x and
x' are equal, which they are not from the above.

So, what is the exact model for Calc's arithmetic?
Which calculations are performed exactly on fundamental arithmetic
operations?

Thanks
Michel
_______________________________________________
LibreOffice mailing list
LibreOffice@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/libreoffice