Paul Tillotson <pntil@shentel.net> writes:
> mod(x, y) is computed as x - trunc(x / y) * y in the mod_var() function
> However, it appears that the division operator itself is rounding up,
This is because div_var rounds its output to the number of fractional
digits specified by select_div_scale, which saith
/*
* The result scale of a division isn't specified in any SQL standard.
* For PostgreSQL we select a result scale that will give at least
* NUMERIC_MIN_SIG_DIGITS significant digits, so that numeric gives a
* result no less accurate than float8; but use a scale not less than
* either input's display scale.
*/
You get the "correct" answer if you force a couple more digits to be
carried by increasing either input's dscale, eg
regression=# select 123456789012345678901234567890.00 % 123;
?column?
----------
117.00
(1 row)
It could be that select_div_scale needs to allow some more slop, or that
that routine is okay but mod_var shouldn't depend on it to select the
intermediate result scale for its internal division. Comments?
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
