i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Michael Franklin <mfranklin@xxxxxxxxxxxxxxx>]

Hello,

I'm experimenting with the realtime PREEMPT_RT patches on a Raspberry Pi 
4 and Raspberry Pi 5.  I'm using 6.1.66-rt19, but have also tested with 
the latest 6.7 kernel and RT patches.

In most of my experiments the realtime kernel improves jitter over the 
stock kernel, but I've discovered that when using i2c, the jitter is 
worse in the realtime kernel than the stock kernel.

It's a little difficult to describe, but can be seen quite clearly in 
this annotated video: 
https://comfiletechdownloads.z12.web.core.windows.net/RT_i2c_jitter.mp4

The only difference between the stock kernel and the realtime kernel is 
that full preemption is enabled in `menuconfig` for the realtime 
kernel.  The kernel is booted with `isolcpus=3` and the program is moved 
to core 3 with `task set -cp 3 $(pidof i2ctest)`.  Also the program is 
scheduled as SCHED_FIFO.

The i2c test program is very simple.  It is written in C/C++, and simply 
sends 2 bytes to an MCP23017 IO expander.  It is basically the following 
C++ pseudocode:

struct sched_param param;
param.sched_priority = 99;
if (sched_setscheduler(0, SCHED_FIFO, &param) != 0)
{
    perror("sched_setscheduler");
    return 1;
}

int fd = open("/dev/i2c-1");
unsigned char data[2];

while(1)
{
    struct i2c_msg messages[] =
    {
        {
            .addr = 0x20,
            .len = 2,
            .buf = data,
        },
    };

    struct i2c_rdwr_ioctl_data payload =
    {
        .msgs = messages,
        .nmsgs = sizeof(messages) / sizeof(messages[0]),
    };

    ioctl(fd, I2C_RDWR, &payload);

    // to avoid requiring `echo -1 > /proc/sys/kernel/sched_rt_runtime_us`
    std::this_thread::sleep_for(1us);
}

The communication works fine, but it's just too jittery in the reatltime 
kernel.

Interestingly, in the stock kernel, `htop` shows that most CPU activity 
is concentrated on core 3 (which is what I expected and preferred), 
while in the realtime kernel, the CPU activity is distributed across all 
cores, despite booting with `isolcpus=3` and running the test program 
with `task set -cp 3 $(pidof i2ctest)` in both kernels.

Q1:  Why is i2c communication is more jittery in the realtime kernel 
than the stock kernel?

Q2:  Why is activity distributed across all cores in the realtime 
kernel, but more concentrated on core 3 in the stock kernel?

Q3:  Is there anything that can be done, either via kernel 
configuration, boot parameters, or something else, that improve the 
jitter in the realtime kernel for this specific use case?

Thank you for your time and consideration,

Mike