Michael Franklin
Software Engineer
COMFILE Technology
82-2-711-2592 ext 510
Skype: MFranklinAtComfile
On 12/13/2023 12:33 PM, Michael Franklin wrote:
On 12/12/2023 9:42 PM, Mike Galbraith wrote:
On Tue, 2023-12-12 at 11:53 +0900, Michael Franklin wrote:
Interestingly, in the stock kernel, `htop` shows that most CPU activity
is concentrated on core 3 (which is what I expected and preferred),
while in the realtime kernel, the CPU activity is distributed across all
cores, despite booting with `isolcpus=3` and running the test program
with `task set -cp 3 $(pidof i2ctest)` in both kernels.
Q1: Why is i2c communication is more jittery in the realtime kernel
than the stock kernel?
I'd speculate it's primarily due to threaded IRQ handling being both
more expensive and preemptible.
Q2: Why is activity distributed across all cores in the realtime
kernel, but more concentrated on core 3 in the stock kernel?
I don't see that. Using isolcpus or not, the test proggy wakes only on
CPU3 (as it had damn well better), and box wide affinity i2c IRQ thread
wakes on CPU0.
Booting the non-rt kernel with 'threadirqs' behaves the same, and I
suspect will jitter about the same should you try it. You're isolating
the test proggy, but for the rt kernel the IRQ thread is left dangling
in the breeze to be perturbed by other IRQ threads or whatnot.
-Mike
Thanks.
I tested the stock kernel with `threadirqs` and indeed the jitter was worse, but the RT kernel still
seemed to be more jittery.
However, based on what you mentioned, I decided to look further interrupts/IRQ behavior, and found
this in /proc/interrupts:
PREEMPT_RT kernel:
CPU0 CPU1 CPU2 CPU3
109: 22625815 0 0 0 rp1_irq_chip 8 Level 1f00074000.i2c
IPI0: 19640 12018430 6870398 5548908 Rescheduling interrupts
IPI1: 713 26295 18104 367 Function call interrupts
Stock Kernel:
CPU0 CPU1 CPU2 CPU3
109: 11129247 0 0 0 rp1_irq_chip 8 Level 1f00074000.i2c
IPI0: 582 620 572 694 Rescheduling interrupts
IPI1: 40061 12360 108946 5475437 Function call interrupts
Stock Kernel with `threadirqs`:
CPU0 CPU1 CPU2 CPU3
109: 21774128 0 0 0 rp1_irq_chip 8 Level 1f00074000.i2c
IPI0: 674 657 687 617 Rescheduling interrupts
IPI1: 58655 127527 21331238 5780380 Function call interrupts
There you can see that, in the PREEMPT_RT kernel, there are a very large number of 'Rescheduling
interrupts' and only a few 'Function call interrupts'. However, in the stock kernel it is the
opposite -- a large number of 'Function call interrupts' and a few 'Rescheduling interrupts'.
Adding `threadirqs` to the stock kenel seemed to just distribute many of the Function call
interrupts over more cores.
Is there anything that can be done about the large number of Rescheduling interrupts in the
PREEMPT_RT kernel?
I was able to reduce the number of Rescheduling interrupts on the isolated core by `echo -1 >
/proc/sys/kernel/sched_rt_runtime_us` but there are still too many Rescheduling interrupts relative
to Function call interrupts on the other 3 cores and the jitter is still pretty bad.
