Re: rounding problems

[Justin <justin@xxxxxxxxxxxxxxx>]

Sam Mason wrote:

On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
  

Sam Mason wrote:
    

What does foxpro use for storing numbers? or is it just that you never
pushed it hard enough for the abstractions to show through.
      

I know i pushed it.  Foxpro  for the most has only  4 basic data types   
Numeric (similar to Posgresql numeric), Boolean, Date, Text aka 
(string)  The foxpro tables supported far more data types but when every 
it was dumped to variable it acted like one of the 4. 
    

I really meant how much did you check the results, or did you accept
that they were correct?

  

Foxpro did not suffer floating point math errors.  I normally used 8 to 
10 points precision.  Foxpro was limited to 15 points of precision  
period.   No more and no less, once you hit that was it.
    

15 places seems very similar to what a 64bit IEEE floating point number
will give you, i.e. a double in C/C++.

  

My problem is we calculate resistance of parts in a Foxpro app that we 
want to move because we want to bring all the custom apps into one 
framework and single database.

Take this calculation  (0.05/30000* 1.0025) which is used to calculate 
parts resistance and Tolerance. (its Ohms Law)  The value returned  from 
C++ = .0000016708 which is wrong
it should be .00000167418.  We just shrank the tolerance on the part we 
make
    

Why are you so sure about the FoxPro result?  I've just checked a few
calculators and get results consistent with your C++ version.

  Justin C: 0.0000016708
  J FoxPro: 0.00000167418
  

this 167418 came of my ti 89 calculator, going back i noticed that i accident rounded it to .00000167 which gives a bad result.

So what i typed in after that point is wrong.  OOPS.

But loosing the 3 will put out of the tolerance sense its the last significant digit needed thats displayed on the measurement devices.  So if the 3 becomes a 4 your out of tolerance.

      My C: 0.000001670833
     bc[1]: 0.0000016708333333333333333333333333333332
     PG[2]: 0.0000016708333333333333336675
  

 Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
  

Foxpro Agrees with what you have  0.00000167083333
the code looks like this

SET DECIMALS TO 15
? ((0.05/30000)* 1.0025)

When i wrote the application like 10 years ago I spent allot time making sure the numbers where correct even doing  some by hand.

If I gotten it wrong there's allot National labs, Universities, Big companies that are generating allot bad results in their QC departments.  

Chced

Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
the math, and as they all agree I'm thinking FoxPro is incorrect!

Here is the foxpro Documentation

Integers or decimal numbers For example, the quantity of items ordered

8 bytes in memory; 1 to 20 bytes in table

- .9999999999E+19 to .9999999999E+20

  Next
I tried doing it accurately (in Haskell if it makes any difference) and
get an answer of 401/240000000 out, which would agree with everything
but FoxPro.  If I calculate the ratio back out for FoxPro I get
401/239520242 which is a little way out.
  

The Documentation from MS says 15 points of precision but the result say 
otherwise.
    

The docs for what? FoxPro or their C compiler?