> -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: Thursday, April 29, 2004 2:31 PM > To: Dann Corbit > Cc: Bruno Wolff III; Paul Tillotson; pgsql-general@postgresql.org > Subject: Re: Arbitrary precision modulo operation > > > "Dann Corbit" <DCorbit@connx.com> writes: > > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > >> Fine. How many is that, exactly? > > > Here is what I would suggest: > > Using the outline I proposed before (starting with a floating point > > divide of DBL_DIG digits of precision), keep doubling the precision > > until the precision is 5 digits larger than either operand. > If the last > > doubling makes the precision larger (quite likely) simply > reduce it to > > the smaller margin. > > And this guarantees a correct answer why? > > AFAIK div_var is already correct per its spec, which is that it > generates an answer rounded to the requested number of digits. > The question at hand is what number of digits to request. > > After thinking about it I don't see any reason that DBL_DIG has > anything to do with a non-surprising answer ... much less > "DBL_DIG + 5" > which seems picked out of the air ... DBL_DIG is there (from float.h) to define the precision of the first operation. After that, the precision doubles each time. 5 was pretty much picked out of the air (it would be good enough to ensure correctness but is overkill). I assumed that you could lose up to one ULP per operation. Do it any way you like. It was just a suggestion. It is clear that mod is not working correctly now, and I was trying to be helpful. ---------------------------(end of broadcast)--------------------------- TIP 8: explain analyze is your friend
