Re: [PATCH v12 09/16] iio: afe: rescale: fix accuracy for small fractional scales

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi!

On 2022-01-09 14:02, Andy Shevchenko wrote:
> On Sat, Jan 8, 2022 at 10:53 PM Liam Beguin <liambeguin@xxxxxxxxx> wrote:
>>
>> The approximation caused by integer divisions can be costly on smaller
>> scale values since the decimal part is significant compared to the
>> integer part. Switch to an IIO_VAL_INT_PLUS_NANO scale type in such
>> cases to maintain accuracy.
> 

...

> ...
> 
>> +               rem2 = *val % (int)tmp;
>> +               *val = *val / (int)tmp;
>> +
>> +               *val2 = rem / (int)tmp;
> 
> Hmm... You divide s64 by 10^9, which means that the maximum value can
> be ~10^10 / 2 (because 2^64-1 ~= 10^19), but this _might_ be bigger
> than 'int' can hold. Can you confirm that tmp can't be so big?
> 

If tmp is so big that it does not fit in 32-bits, that is indeed a
problem, and it means that the scale has overflowed. However, the
problem with the scale overflowing very much existed prior to this
series. Doing something about that overflow is not strictly related
to improving the accuracy for small scale factors.

Cheers,
Peter



[Index of Archives]     [Device Tree Compilter]     [Device Tree Spec]     [Linux Driver Backports]     [Video for Linux]     [Linux USB Devel]     [Linux PCI Devel]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [XFree86]     [Yosemite Backpacking]


  Powered by Linux