Search Postgresql Archives

Re: rounding problems

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, May 14, 2008 at 11:47:52AM -0400, Justin wrote:
> I have forgotten how much i hate C++

What we're talking about doesn't have much to do with C++, it's floating
point maths in general.

> Its not doing what you say it would but it did do other odd ball 
> things.  I miss my foxpro :-(. 

What does foxpro use for storing numbers? or is it just that you never
pushed it hard enough for the abstractions to show through.

> Plus its not holding 15 precision points

after changing the output to be:

  printf("%.10f %.10f\n", d, d-c);

I get:

  100000000.0999999940 0.0999999940
  100000000.0100000054 0.0100000054
  100000000.0010000020 0.0010000020
  100000000.0001000017 0.0001000017
  100000000.0000099987 0.0000099987
  100000000.0000009984 0.0000009984
  100000000.0000001043 0.0000001043
  100000000.0000000149 0.0000000149
  100000000.0000000000 0.0000000000

Which looks reasonable.  Remember that floating point numbers store
their state in base two, not base ten.  All of those numbers look good
to 15 decimal digits.


  Sam


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Postgresql Jobs]     [Postgresql Admin]     [Postgresql Performance]     [Linux Clusters]     [PHP Home]     [PHP on Windows]     [Kernel Newbies]     [PHP Classes]     [PHP Books]     [PHP Databases]     [Postgresql & PHP]     [Yosemite]
  Powered by Linux