Hi Michel, for example look for ScAdd in https://opengrok.libreoffice.org/xref/core/sc/source/core/tool/interpr5.cxx and for the rtl::math things in https://opengrok.libreoffice.org/xref/core/include/rtl/math.hxx Kind regards Regina Michel Onoff schrieb am 03-Dec-20 um 13:48:
Hi, I would like to understand how simple arithmetic (+ - * /) works in Calc. I kind of guess that the underlying internal number representation is IEEE 754 double (64 bit). I also guess that when a double is shown on the sheet it is approximated by a decimal with at most 15 significand digits. To reproduce the behavior below, use scientific notation with 20 digits or more. I have two slightly different number x and x'. x is 2^-49 (a formula) while x' is 1.77635683940025E-15 (a literal). Their decimal representation appear equal on the sheet, but they are, indeed, slightly different internally. You can set them apart as follows: y = x - 1.7763568394002E-15 y' = x' - 1.7763568394002E-15 that is, by subtracting the same number from x and x'. y and y' appear differently on the sheet, meaning that x and x' are different internally to start with. However, x - x' is exactly 0. In IEEE 754 arithmetic, two numbers are equal if and only if their difference is 0. That would mean that x and x' are equal, which they are not from the above. So, what is the exact model for Calc's arithmetic? Which calculations are performed exactly on fundamental arithmetic operations? Thanks Michel _______________________________________________ LibreOffice mailing list LibreOffice@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/libreoffice
_______________________________________________ LibreOffice mailing list LibreOffice@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/libreoffice
