Here is my new system:
Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz
C422 chipset
Centos 8
I've compiled and built two versions of the realtime kernel to take
the place of the stock Centos kernel:
4.19.103-rt42
and
5.4.17-rt9
For both kernels, I run cyclictest:
./cyclictest -p 98 --smp -b 600 -f -m
The worst case latencies approach 2 milliseconds at times.
I have disabled SpeedStep in the BIOS.
I have also experimented with the following additional kernel boot
parameters for both kernels:
noibrs noibpb nospectre_v2 nospectre_v1 l1tf=off
nospec_store_bypass_disable no_stf_barrier mds=off mitigations=off
nosoftlockup intel_idle.max_cstate=0 mce=ignore_ce
processor.max_cstate=0
These parameters didn't solve the problem.
The trace results show that the unexpected large latency seems to
occur in the following ways:
llvmpipe-9436 3d...... 4901974us!: switch_fpu_return
<-prepare_exit_to_usermode
llvmpipe-9436 3d...... 4902845us : smp_apic_timer_interrupt
<-apic_timer_interrupt
or
Xorg-8825 3....... 4905576us!: kfree <-__audit_syscall_exit
<idle>-0 2d...1.. 4905876us : smp_apic_timer_interrupt
<-apic_timer_interrupt
or
<idle>-0 2d...1.. 4905910us!: mwait_idle <-default_idle_call
<idle>-0 0d...1.. 4906049us : smp_apic_timer_interrupt
<-apic_timer_interrupt
or
llvmpipe-9435 1d...... 4917323us!: rcu_irq_exit <-irq_exit
llvmpipe-9438 3d...... 4917845us : smp_apic_timer_interrupt
<-apic_timer_interrupt
I do see NMIs occurring on the system, although not all latency events
seem to correlate with an increment in the NMI counter in
/proc/interrupts.
I would greatly appreciate any advice on what I should do to trace the
problem with this new system. I can send my .config files if needed.
CONFIG_PREEMPT_RT_FULL=y is set.
Thanks!
Jan
