On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
My problem is we calculate resistance of parts in a Foxpro app
that we
want to move because we want to bring all the custom apps into one
framework and single database.
Take this calculation (0.05/30000* 1.0025) which is used to
calculate
parts resistance and Tolerance. (its Ohms Law) The value
returned from
C++ = .0000016708 which is wrong it should be .00000167418.
Why are you so sure about the FoxPro result? I've just checked a few
calculators and get results consistent with your C++ version.
Justin C: 0.0000016708
J FoxPro: 0.00000167418
My C: 0.000001670833
bc[1]: 0.0000016708333333333333333333333333333332
PG[2]: 0.0000016708333333333333336675
Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
the math, and as they all agree I'm thinking FoxPro is incorrect!
Next
I tried doing it accurately (in Haskell if it makes any difference)
and
get an answer of 401/240000000 out, which would agree with everything
but FoxPro. If I calculate the ratio back out for FoxPro I get
401/239520242 which is a little way out.
I'll add my Casio scientific calculator to the list, which also gives
the non-FoxPro result.
We can also write this "exactly" as:
(0.05 * 1.0025 / 3) * 10^-5
= (5 * 10025 / 3) * 10^-11
= (50,125 / 3) * 0.00000000001
= (16,708 + 1/3) * 0.00000000001
which you can verify in your head, so the correct answer is clear.
Is it possible that the FoxPro calculation isn't what you describe?
If you use the tolerance 1.0045, a single digit off, then you get
0.00000167417, very close to your result.
-- Andy
