Example:

`The 5V sensor with an input of precisely 5V should give an input MSB`

`register value of 192.`

Using the "full scale" ratios: 192 * 6640 / 256 = 4980 Using the 3/4 scale ratios: 192 * 5000 / 192 = 5000 Error calculation: (5000 - 4980) / 5000 = 0.4%

`This corresponds to a 1-bit error in the 8 bit voltages, or a 3-bit`

`error in the 10 bit voltages, effectively giving only 7-bit accuracy in`

`the calculated values. An unnecessary error I think you'll agree.`

`Although the full scale values could probably be "tweaked" to make them`

`slightly better, the rounding issues inherent with integer arithmetic`

`make it difficult to get right on the button near the nominal voltages,`

`where highest accuracy is appropriate. This is easily avoided by using`

`the simple 3/4 scale ratios given in the data sheet.`

On 07/03/10 10:44, Jean Delvare wrote:

On Sun, 07 Mar 2010 10:20:13 +0000, Ken Milmore wrote:Regarding the scale factors, I think you'll find that those given for the 2.5V, 3.3V and 5V ranges are far enough wrong introduce a 2-bit error in the 10-bit values. The driver goes to some trouble to get the full 10 bit precision by combining both the MSB and LSB registers, then throws that precision away in the calculations! You really do need to use the 3/4 scale voltage values, or some equivalent rational scaling, to get the right answers here. That's why Andigilog provide them in the data sheet! :-)Whether you use the full scale or 3/4 scale voltage values as the base for the calculation doesn't matter, both can work as long as the calculation is consistent. What really matters is to multiply before you divide.I will try and get a patch together as soon as time permits.Great, thanks.

_______________________________________________ lm-sensors mailing list lm-sensors@xxxxxxxxxxxxxx http://lists.lm-sensors.org/mailman/listinfo/lm-sensors