Search Postgresql Archives

Re: rounding problems

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On May 14, 3:27 pm, s...@xxxxxxxxxxxxx (Sam Mason) wrote:
> On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
> > Sam Mason wrote:
> > >What doesfoxprouse for storing numbers? or is it just that you never
> > >pushed it hard enough for the abstractions to show through.
>
> > I know i pushed it.  Foxpro for the most has only  4 basic data types
> > Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
> > (string)  Thefoxprotables supported far more data types but when every
> > it was dumped to variable it acted like one of the 4.
>
> I really meant how much did you check the results, or did you accept
> that they were correct?
>
> >Foxprodid not suffer floating point math errors.  I normally used 8 to
> > 10 points precision.  Foxprowas limited to 15 points of precision
> > period.   No more and no less, once you hit that was it.
>
> 15 places seems very similar to what a 64bit IEEE floating point number
> will give you, i.e. a double in C/C++.
>
> > My problem is we calculate resistance of parts in aFoxproapp that we
> > want to move because we want to bring all the custom apps into one
> > framework and single database.
>
> > Take this calculation  (0.05/30000* 1.0025) which is used to calculate
> > parts resistance and Tolerance. (its Ohms Law)  The value returned  from
> > C++ = .0000016708 which is wrong
> > it should be .00000167418.  We just shrank the tolerance on the part we
> > make
>
> Why are you so sure about theFoxProresult?  I've just checked a few
> calculators and get results consistent with your C++ version.
>
>   Justin C: 0.0000016708
>   JFoxPro: 0.00000167418
>       My C: 0.000001670833
>      bc[1]: 0.0000016708333333333333333333333333333332
>      PG[2]: 0.0000016708333333333333336675
>  Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
>
> Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
> the math, and as they all agree I'm thinkingFoxProis incorrect!  Next
> I tried doing it accurately (in Haskell if it makes any difference) and
> get an answer of 401/240000000 out, which would agree with everything
> butFoxPro.  If I calculate the ratio back out forFoxProI get
> 401/239520242 which is a little way out.
>
> > The Documentation from MS says 15 points of precision but the result say
> > otherwise.
>
> The docs for what?FoxProor their C compiler?
>
> If you meanFoxPro, I think this is another case of MS screwing up.
>
> > I'm glad You and others are taking the time to explain to me
> > the odd results before i get into redoing that application.
>
> Welcome to the PG community, lots of people to get interested in lots of
> things!
>
> > Why oh Why did MS killFoxpro. :'(   I understood it, knew its quirks
> > and it worked very well with Postgresql
>
> Are you sure you want to stay with it if its answers are wrong?
>
>   Sam

*********************************************************************************
This is fun, at 0400 AM.  I enjoy reading  Experts having serious fun!

VFP 6.0, using my defaults
? (0.05/30000* 1.00250000000000000)
displays  "0l.000001670833333000"

SET DECIMALS TO 15
? ((0.05/30000)* 1.0025)
displays "0.000001670833333"

and a frivolous example:
SET DECIMALS TO 18
? ((0.050000/30000.00000000)* 1.0025000000000000)
displays "0.000001670833333000"

Anybody tried to reckon this math
the way we used to do it with a Slide-Rule ???
(In VFP of course)

glene77is


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Postgresql Jobs]     [Postgresql Admin]     [Postgresql Performance]     [Linux Clusters]     [PHP Home]     [PHP on Windows]     [Kernel Newbies]     [PHP Classes]     [PHP Books]     [PHP Databases]     [Postgresql & PHP]     [Yosemite]
  Powered by Linux