Hi Marco, Peter, I got a report that 'perf test sigtrap' test failed on a PREEMPT_RT_FULL kernel, one that had up to: commit 97ba62b278674293762c3d91f724f1bb922f04e0 Author: Marco Elver <elver@xxxxxxxxxx> Date: Thu Apr 8 12:36:01 2021 +0200 perf: Add support for SIGTRAP on perf events It failed with no sigtrap delivered, none of the nr-threads + interactions (15000). Then I tried backporting up to the perf subsystem refactorings, and then to what is in https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/log/?h=linux-6.4.y-rt, but in both cases I ended up with the splat at the end of this message. I'll continue investigating, but thought it would be a good time to report it. - Arnaldo [ 52.848925] BUG: scheduling while atomic: perf/6549/0x00000002 [ 52.848925] BUG: scheduling while atomic: perf/6547/0x00000002 [ 52.848931] Modules linked in: [ 52.848931] Modules linked in: [ 52.848932] nft_fib_inet [ 52.848931] BUG: scheduling while atomic: perf/6548/0x00000002 [ 52.848932] nft_fib_inet nft_fib_ipv4 <SNIP> [ 52.849055] Preemption disabled at: [ 52.849055] Preemption disabled at: [ 52.849055] Preemption disabled at: [ 52.849056] [<0000000000000000>] 0x0 [ 52.849056] [<0000000000000000>] 0x0 [ 52.849056] [<0000000000000000>] 0x0 [ 52.849061] CPU: 2 PID: 6549 Comm: perf Not tainted 6.4.0-rt6+ #2 [ 52.849064] Hardware name: LENOVO 427623U/427623U, BIOS 8BET45WW (1.25 ) 05/18/2011 [ 52.849066] Call Trace: [ 52.849069] <TASK> [ 52.849071] dump_stack_lvl+0x33/0x50 [ 52.849077] __schedule_bug+0x9a/0xb0 [ 52.849082] schedule_debug.constprop.0+0x10f/0x140 [ 52.849086] __schedule+0x50/0x6c0 [ 52.849090] ? _raw_spin_lock+0x13/0x40 [ 52.849093] ? task_blocks_on_rt_mutex.constprop.0.isra.0+0x1b2/0x440 [ 52.849096] schedule_rtlock+0x1e/0x40 [ 52.849099] rtlock_slowlock_locked+0xf2/0x360 [ 52.849102] ? perf_ctx_enable+0x44/0x60 [ 52.849106] rt_spin_lock+0x41/0x60 [ 52.849109] do_send_sig_info+0x32/0xb0 [ 52.849112] send_sig_perf+0x70/0x90 [ 52.849116] perf_pending_task+0xb1/0xd0 [ 52.849119] task_work_run+0x59/0x90 [ 52.849123] exit_to_user_mode_loop+0x128/0x130 [ 52.849128] exit_to_user_mode_prepare+0xbd/0xd0 [ 52.849131] irqentry_exit_to_user_mode+0x5/0x30 [ 52.849135] asm_sysvec_irq_work+0x16/0x20 [ 52.849138] RIP: 0033:0x55aa823bed63 [ 52.849141] Code: 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 e8 e5 03 f5 ff 4c 89 e7 48 89 c3 e8 da 1c f5 ff f0 01 1d a3 f4 91 00 8b 05 a5 f4 91 00 <83> f8 01 7e 1f 89 d9 31 d2 0f 1f 40 00 f0 01 0d 89 f4 91 00 8b 05 [ 52.849143] RSP: 002b:00007f31341ccdb0 EFLAGS: 00000206 [ 52.849145] RAX: 0000000000000bb8 RBX: 0000000000001995 RCX: 00007f31369aab44 [ 52.849147] RDX: 0000000000000000 RSI: 0000000000000081 RDI: 00007fffcdbf8384 [ 52.849148] RBP: 00007f31341ccdd0 R08: 00007fffcdbf8380 R09: 0000000000000006 [ 52.849150] R10: 0000000000000000 R11: 0000000000000286 R12: 00007fffcdbf8380 [ 52.849151] R13: 000000000000000d R14: 00007f31369ac530 R15: 0000000000000000 [ 52.849156] </TASK> [ 52.849157] CPU: 3 PID: 6547 Comm: perf Not tainted 6.4.0-rt6+ #2 [acme@nine linux]$ git log --oneline -10 d37d728e9a66 (HEAD, tag: v6.4-rt6, linux-rt-devel/linux-6.4.y-rt) v6.4-rt6 4d1139baae8b mm/page_alloc: Use write_seqlock_irqsave() instead write_seqlock() + local_irq_save(). dc93c1f07d48 seqlock: Do the lockdep annotation before locking in do_write_seqcount_begin_nested() a3f6be6e5353 printk: Check only for migration in printk_deferred_*(). 6dc15eb2a631 bpf: Remove in_atomic() from bpf_link_put(). 0ccbab373cd7 (tag: v6.4-rt5) v6.4-rt5 786cdf91804d Merge tag 'v6.4' into linux-6.4.y-rt 6995e2de6891 (tag: v6.4, linux-rt-devel/master, linux-rt-devel/linux-6.4.y) Linux 6.4 e3b2e2c14bcc Merge tag 'i2c-for-6.4-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux 547cc9be86f4 Merge tag 'perf_urgent_for_v6.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip [acme@nine linux]$ [acme@nine linux]$ uname -a Linux nine 6.4.0-rt6+ #2 SMP PREEMPT_RT Sat Jul 22 08:52:53 -03 2023 x86_64 x86_64 x86_64 GNU/Linux [acme@nine linux]$ cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="9.2 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.2" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.2 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.2" [acme@nine linux]$