Hi Ken, On Sun, 07 Mar 2010 11:30:13 +0000, Ken Milmore wrote: >>Example:>>The 5V sensor with an input of precisely 5V should give an input MSB>register value of 192.>>Using the "full scale" ratios:>>192 * 6640 / 256 = 4980>>Using the 3/4 scale ratios:>>192 * 5000 / 192 = 5000>>Error calculation:>>(5000 - 4980) / 5000 = 0.4%>>This corresponds to a 1-bit error in the 8 bit voltages, or a 3-bit>error in the 10 bit voltages, effectively giving only 7-bit accuracy in>the calculated values. An unnecessary error I think you'll agree.>>Although the full scale values could probably be "tweaked" to make them>slightly better, the rounding issues inherent with integer arithmetic>make it difficult to get right on the button near the nominal voltages,>where highest accuracy is appropriate. This is easily avoided by using>the simple 3/4 scale ratios given in the data sheet.No, you're off track, sorry. The only problem in the driver's formula is that 6640 was computed as 5000 * 255 / 192, but the formula divides by 256 instead of 255. This is the inconsistency that leads to the 0.4% error (1/256 = 0.4%). Just use consistent numbers and it will work fine. The rounding issues inherent to integer arithmetic induce much smaller errors, down to a level you won't notice them. In the case above, the max error would be 0.5/6640 = 0.0075%... Way below the resolution and accuracy of the sensors. Now, if you insist on using the 3/4 scale values as a basis, really, I don't care. All I'm saying is this this design change is not _required_ to make the driver return correct values. -- Jean Delvare _______________________________________________ lm-sensors mailing list lm-sensors@xxxxxxxxxxxxxx http://lists.lm-sensors.org/mailman/listinfo/lm-sensors