Re: i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





Michael Franklin
Software Engineer
COMFILE Technology
82-2-711-2592 ext 510
Skype: MFranklinAtComfile
On 12/13/2023 12:33 PM, Michael Franklin wrote:
On 12/12/2023 9:42 PM, Mike Galbraith wrote:
On Tue, 2023-12-12 at 11:53 +0900, Michael Franklin wrote:

Interestingly, in the stock kernel, `htop` shows that most CPU activity
is concentrated on core 3 (which is what I expected and preferred),
while in the realtime kernel, the CPU activity is distributed across all
cores, despite booting with `isolcpus=3` and running the test program
with `task set -cp 3 $(pidof i2ctest)` in both kernels.

Q1:  Why is i2c communication is more jittery in the realtime kernel
than the stock kernel?

I'd speculate it's primarily due to threaded IRQ handling being both
more expensive and preemptible.

Q2:  Why is activity distributed across all cores in the realtime
kernel, but more concentrated on core 3 in the stock kernel?

I don't see that.  Using isolcpus or not, the test proggy wakes only on
CPU3 (as it had damn well better), and box wide affinity i2c IRQ thread
wakes on CPU0.

Booting the non-rt kernel with 'threadirqs' behaves the same, and I
suspect will jitter about the same should you try it.  You're isolating
the test proggy, but for the rt kernel the IRQ thread is left dangling
in the breeze to be perturbed by other IRQ threads or whatnot.

    -Mike

Thanks.

I tested the stock kernel with `threadirqs` and indeed the jitter was worse, but the RT kernel still seemed to be more jittery.

However, based on what you mentioned, I decided to look further interrupts/IRQ behavior, and found this in /proc/interrupts:

PREEMPT_RT kernel:
            CPU0       CPU1       CPU2       CPU3
109:   22625815          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:     19640   12018430    6870398    5548908       Rescheduling interrupts
IPI1:       713      26295      18104        367       Function call interrupts

Stock Kernel:
            CPU0       CPU1       CPU2       CPU3
109:   11129247          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:       582        620        572        694       Rescheduling interrupts
IPI1:     40061      12360     108946    5475437       Function call interrupts

Stock Kernel with `threadirqs`:
            CPU0       CPU1       CPU2       CPU3
109:   21774128          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:       674        657        687        617       Rescheduling interrupts
IPI1:     58655     127527   21331238    5780380       Function call interrupts

There you can see that, in the PREEMPT_RT kernel, there are a very large number of 'Rescheduling interrupts' and only a few 'Function call interrupts'.  However, in the stock kernel it is the opposite -- a large number of 'Function call interrupts' and a few 'Rescheduling interrupts'.

Adding `threadirqs` to the stock kenel seemed to just distribute many of the Function call interrupts over more cores.

Is there anything that can be done about the large number of Rescheduling interrupts in the PREEMPT_RT kernel?

I was able to reduce the number of Rescheduling interrupts on the isolated core by `echo -1 > /proc/sys/kernel/sched_rt_runtime_us` but there are still too many Rescheduling interrupts relative to Function call interrupts on the other 3 cores and the jitter is still pretty bad.





[Index of Archives]     [RT Stable]     [Kernel Newbies]     [IDE]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux ATA RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux