Hi Michel, On Thursday, 2020-12-03 13:48:25 +0100, Michel Onoff wrote: > I would like to understand how simple arithmetic (+ - * /) works in Calc. > I kind of guess that the underlying internal number representation is > IEEE 754 double (64 bit). Yes. > I also guess that when a double is shown on the sheet it is approximated > by a decimal with at most 15 significand digits. Yes. > To reproduce the behavior below, use scientific notation with 20 digits > or more. Note that with increasing digits beyond 15 significands you currently will not gain anything except additional zeros. > I have two slightly different number x and x'. > x is 2^-49 (a formula) while x' is 1.77635683940025E-15 (a literal). > Their decimal representation appear equal on the sheet, but they are, > indeed, slightly different internally. You can set them apart as follows: > > y = x - 1.7763568394002E-15 > y' = x' - 1.7763568394002E-15 > > that is, by subtracting the same number from x and x'. > y and y' appear differently on the sheet, meaning that x and x' are > different internally to start with. Yes. > However, x - x' is exactly 0. In IEEE 754 arithmetic, two numbers are > equal if and only if their difference is 0. That would mean that x and > x' are equal, which they are not from the above. The Calc + and - operators try to cater for numeric inaccuracies and tie to zero in some ranges, otherwise the famous example of 0.3 - 0.2 - 0.1 would not be 0.0 but -2.77555756156289E-17 instead. If you want the "raw" subtraction of your two values you can check with =RAWSUBTRACT(2^-49;1.77635683940025E-015) that yields 3.94430452610506E-31 The double to string conversion needs further improvement to be able to more precisely represent values in decimals so could actually convert 2^-49 to 1.7763568394002505E-15 Eike -- GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918 630B 6A6C D5B7 6563 2D3A
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ LibreOffice mailing list LibreOffice@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/libreoffice
