I built 5.15.28-rt35 #2 SMP PREEMPT_RT with “Fully Preemptible Kernel" option and otherwise using .config from a stock Ubuntu 20.04 5.4.0 kernel. Quick question: I see scheduling/wake_up latencies around 800 usec and this is confirmed by cyclictest. Is there a link that can point me to systematic steps I can take to trace down if my problem is a wrong config param (un-needed, hurtful debug option) or this is best that I can do (on good modern HW.)? I am hoping that with PREEMPT-RT on an unloaded system running at SCHED_FIFO prio 99 I can get max latencies of <= 100 usec? Is this possible? Details: node-0> grep -i preempt /boot/config-5.15.28-rt35 CONFIG_HAVE_PREEMPT_LAZY=y CONFIG_PREEMPT_LAZY=y # CONFIG_PREEMPT_NONE is not set # CONFIG_PREEMPT_VOLUNTARY is not set # CONFIG_PREEMPT is not set CONFIG_PREEMPT_RT=y CONFIG_PREEMPT_COUNT=y CONFIG_PREEMPTION=y CONFIG_PREEMPT_RCU=y CONFIG_HAVE_PREEMPT_DYNAMIC=yA CONFIG_PREEMPT_NOTIFIERS=y # CONFIG_DEBUG_PREEMPT is not set # CONFIG_PREEMPT_TRACER is not set # CONFIG_PREEMPTIRQ_DELAY_TEST is not set node-0> grep -i debug /boot/config-5.15.28-rt35 |grep =y CONFIG_SLUB_DEBUG=y CONFIG_IOSF_MBI_DEBUG=y CONFIG_X86_DEBUGCTLMSR=y CONFIG_PM_DEBUG=y CONFIG_PM_ADVANCED_DEBUG=y CONFIG_PM_SLEEP_DEBUG=y CONFIG_ACPI_DEBUGGER=y CONFIG_ACPI_DEBUGGER_USER=y CONFIG_ACPI_DEBUG=y CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_BLK_DEBUG_FS=y CONFIG_BLK_DEBUG_FS_ZONED=y CONFIG_NFS_DEBUG=y CONFIG_SUNRPC_DEBUG=y CONFIG_DYNAMIC_DEBUG=y CONFIG_DYNAMIC_DEBUG_CORE=y CONFIG_DEBUG_BUGVERBOSE=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_DWARF4=y CONFIG_DEBUG_FS=y CONFIG_DEBUG_FS_ALLOW_ALL=y CONFIG_ARCH_HAS_EARLY_DEBUG=y CONFIG_DEBUG_KERNEL=y CONFIG_DEBUG_MISC=y CONFIG_ARCH_HAS_DEBUG_WX=y CONFIG_DEBUG_WX=y CONFIG_HAVE_DEBUG_KMEMLEAK=y CONFIG_ARCH_HAS_DEBUG_VM_PGTABLE=y CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y CONFIG_SCHED_DEBUG=y CONFIG_LOCK_DEBUGGING_SUPPORT=y CONFIG_X86_DEBUG_FPU=y My test involves using "nanosleep(x)" with different values of "x" and examining stats of actual sleep durations. I see: #nanosleep_duration minimum: maximum: mean: num_points: Range 100 103 934 108.635 100000 831 500 504 1377 543.272 100000 873 1000 1010 1870 1053.24 100000 860 2000 2010 2872 2052.46 100000 862 4000 4011 4906 4046.66 100000 895 10000 10013 10738 10067.3 100000 725 I next ran cyclictest which mostly agrees with the ~700-800 max latency I see in my own tests: node-0> sudo ./cyclictest --mlockall --smp --priority=80 --interval=1000 --distance=0 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 2.84 0.86 0.62 1/551 8688 T: 0 ( 8655) P:80 I:1000 C: 6656 Min: 3 Act: 3 Avg: 4 Max: 24 T: 1 ( 8656) P:80 I:1000 C: 6650 Min: 3 Act: 4 Avg: 5 Max: 68 T: 2 ( 8657) P:80 I:1000 C: 6645 Min: 3 Act: 4 Avg: 4 Max: 24 T: 3 ( 8658) P:80 I:1000 C: 6640 Min: 3 Act: 4 Avg: 4 Max: 28 T: 4 ( 8659) P:80 I:1000 C: 6635 Min: 3 Act: 3 Avg: 4 Max: 27 T: 5 ( 8660) P:80 I:1000 C: 6629 Min: 3 Act: 3 Avg: 4 Max: 26 T: 6 ( 8661) P:80 I:1000 C: 6625 Min: 3 Act: 3 Avg: 4 Max: 25 T: 7 ( 8662) P:80 I:1000 C: 6620 Min: 3 Act: 3 Avg: 4 Max: 98 T: 8 ( 8663) P:80 I:1000 C: 6615 Min: 3 Act: 4 Avg: 5 Max: 300 T: 9 ( 8664) P:80 I:1000 C: 6610 Min: 3 Act: 3 Avg: 5 Max: 610 T:10 ( 8665) P:80 I:1000 C: 6606 Min: 3 Act: 4 Avg: 4 Max: 16 T:11 ( 8666) P:80 I:1000 C: 6601 Min: 3 Act: 3 Avg: 5 Max: 319 T:12 ( 8667) P:80 I:1000 C: 6596 Min: 3 Act: 5 Avg: 5 Max: 696 T:13 ( 8668) P:80 I:1000 C: 6592 Min: 3 Act: 4 Avg: 5 Max: 149 T:14 ( 8669) P:80 I:1000 C: 6587 Min: 3 Act: 3 Avg: 5 Max: 633 T:15 ( 8670) P:80 I:1000 C: 6583 Min: 3 Act: 3 Avg: 4 Max: 159 T:16 ( 8671) P:80 I:1000 C: 6578 Min: 3 Act: 3 Avg: 5 Max: 661 T:17 ( 8672) P:80 I:1000 C: 6574 Min: 3 Act: 4 Avg: 5 Max: 170 T:18 ( 8673) P:80 I:1000 C: 6569 Min: 3 Act: 3 Avg: 4 Max: 13 T:19 ( 8674) P:80 I:1000 C: 6565 Min: 3 Act: 4 Avg: 4 Max: 348 T:20 ( 8675) P:80 I:1000 C: 6561 Min: 3 Act: 3 Avg: 4 Max: 23 T:21 ( 8676) P:80 I:1000 C: 6556 Min: 3 Act: 4 Avg: 5 Max: 635 T:22 ( 8677) P:80 I:1000 C: 6552 Min: 3 Act: 4 Avg: 5 Max: 349 T:23 ( 8678) P:80 I:1000 C: 6548 Min: 3 Act: 3 Avg: 4 Max: 16 T:24 ( 8679) P:80 I:1000 C: 6543 Min: 3 Act: 4 Avg: 6 Max: 610 T:25 ( 8680) P:80 I:1000 C: 6539 Min: 3 Act: 3 Avg: 4 Max: 99 T:26 ( 8681) P:80 I:1000 C: 6534 Min: 3 Act: 4 Avg: 4 Max: 547 T:27 ( 8682) P:80 I:1000 C: 6530 Min: 3 Act: 3 Avg: 5 Max: 157 T:28 ( 8683) P:80 I:1000 C: 6526 Min: 3 Act: 4 Avg: 4 Max: 26 T:29 ( 8684) P:80 I:1000 C: 6521 Min: 3 Act: 4 Avg: 5 Max: 382 T:30 ( 8685) P:80 I:1000 C: 6517 Min: 3 Act: 3 Avg: 4 Max: 17 T:31 ( 8686) P:80 I:1000 C: 6512 Min: 3 Act: 3 Avg: 5 Max: 639 My HW is a Dell d430 on which /proc/cpuinfo shows 32 processors of type: processor : 31 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz stepping : 2 microcode : 0x46 cpu MHz : 1200.000 cache size : 20480 KB Any directions to how one goes about reducing my max latencies will be much appreciated.
