Hi Frederic Weisbecker and kernel experts, Platform: x86 platforms There is "sys_perf_event_open" soft lockup BUG in v6.3-rc2 kernel in guest. I saw previous fixed patch has been merged into v6.3-rc2 kernel: " commit 28319d6dc5e2ffefa452c2377dd0f71621b5bff0 rcu-tasks: Fix synchronize_rcu_tasks() VS zap_pid_ns_processes() " I'm not sure if there are other new issues, the bisect bad commit is: " c597bfddc9e9e8a63817252b67c3ca0e544ace26 sched: Provide Kconfig support for default dynamic preempt mode " It's just suspected commit, because reverted the above commit on top of v6.3-rc2 and made kernel failed, could not double confirm for this issue. All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230316_062127_sys_perf_event_open Reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230316_062127_sys_perf_event_open/repro.c Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230316_062127_sys_perf_event_open/kconfig_origin v6.3-rc2 issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230316_062127_sys_perf_event_open/eeac8ede17557680855031c6f305ece2378af326_dmesg.log Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230316_062127_sys_perf_event_open/bisect_info.log " [ 24.580606] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=330 'systemd' [ 64.563136] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [repro:511] [ 64.568596] Modules linked in: [ 64.570087] irq event stamp: 640964 [ 64.571259] hardirqs last enabled at (640963): [<ffffffff8107509c>] __text_poke+0x2ec/0x4e0 [ 64.573204] hardirqs last disabled at (640964): [<ffffffff82fff863>] sysvec_apic_timer_interrupt+0x13/0xe0 [ 64.575281] softirqs last enabled at (495656): [<ffffffff83024a4c>] __do_softirq+0x31c/0x49c [ 64.577257] softirqs last disabled at (495631): [<ffffffff8112b5a4>] irq_exit_rcu+0xc4/0x100 [ 64.579275] CPU: 0 PID: 511 Comm: repro Not tainted 6.3.0-rc2-eeac8ede1755+ #1 [ 64.581032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 64.583432] RIP: 0010:__text_poke+0x2f2/0x4e0 [ 64.584873] Code: 06 00 20 00 00 e8 0e f0 07 00 48 81 7d b8 60 49 07 81 0f 84 79 01 00 00 48 83 7d 98 00 74 0b e8 24 f3 2c 00 fb 0f 1f 44 00 00 <48> 8b 7d c8 e8 85 dc fa 01 4c 89 f8 48 8b 4d d0 65 48 33 0c 25 28 [ 64.588300] RSP: 0018:ffffc90000f4fa20 EFLAGS: 00000206 [ 64.590112] RAX: 000000000009c7c3 RBX: 0000000000000000 RCX: 0000000000000006 [ 64.591863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 [ 64.593599] RBP: ffffc90000f4fa90 R08: 0000000000000001 R09: 0000000000000001 [ 64.595348] R10: ffffc90000f4f790 R11: 0000000000000001 R12: ffff88800fe29980 [ 64.597090] R13: 0000000000000001 R14: 0000000000000da9 R15: ffffffff82263da5 [ 64.598831] FS: 00007f7680806740(0000) GS:ffff88807dc00000(0000) knlGS:0000000000000000 [ 64.600704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 64.602240] CR2: 00007fff176ab6a8 CR3: 000000000e6d2004 CR4: 0000000000770ef0 [ 64.604018] PKRU: 55555554 [ 64.605106] Call Trace: [ 64.606177] <TASK> [ 64.607930] ? __pfx_text_poke_memcpy+0x10/0x10 [ 64.609796] ? drm_buddy_fini+0x5/0xe0 [ 64.611842] text_poke+0x3a/0x60 [ 64.613674] text_poke_bp_batch+0x172/0x310 [ 64.615776] ? drm_atomic_helper_page_flip_target+0x4/0x170 [ 64.618053] text_poke_queue+0x93/0xb0 [ 64.620108] ftrace_replace_code+0x12a/0x1b0 [ 64.622526] ftrace_modify_all_code+0x1b9/0x2a0 [ 64.624733] arch_ftrace_update_code+0xd/0x20 [ 64.626307] ftrace_startup_enable+0x67/0xa0 [ 64.628155] ftrace_startup+0x124/0x200 [ 64.630197] register_ftrace_function_nolock+0x43/0x90 [ 64.632164] register_ftrace_function+0x1eb/0x280 [ 64.634074] ? __sanitizer_cov_trace_switch+0x57/0xa0 [ 64.636519] perf_ftrace_event_register+0xcd/0xf0 [ 64.638494] perf_trace_event_init+0x98/0x4b0 [ 64.640836] perf_trace_init+0xde/0x170 [ 64.642951] perf_tp_event_init+0x60/0xa0 [ 64.644800] perf_try_init_event+0x88/0x280 [ 64.647230] perf_event_alloc+0xe25/0x1c00 [ 64.649236] ? perf_event_alloc+0x5/0x1c00 [ 64.652094] __do_sys_perf_event_open+0x3b6/0x1910 [ 64.658662] __x64_sys_perf_event_open+0x2f/0x40 [ 64.660485] do_syscall_64+0x3b/0x90 [ 64.662126] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 64.663729] RIP: 0033:0x7f768092b59d [ 64.665179] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d3 08 0d 00 f7 d8 64 89 01 48 [ 64.668694] RSP: 002b:00007fff176ab6a8 EFLAGS: 00000246 ORIG_RAX: 000000000000012a [ 64.670924] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f768092b59d [ 64.672718] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 00000000200000c0 [ 64.674495] RBP: 00007fff176ab6c0 R08: 0000000000000000 R09: 00007fff176ab6c0 [ 64.676284] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000401180 [ 64.678081] R13: 00007fff176ab7e0 R14: 0000000000000000 R15: 0000000000000000 [ 64.682881] </TASK> [ 64.683972] Kernel panic - not syncing: softlockup: hung tasks [ 64.685592] CPU: 0 PID: 511 Comm: repro Tainted: G L 6.3.0-rc2-eeac8ede1755+ #1 [ 64.687621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014 [ 64.690078] Call Trace: [ 64.691204] <IRQ> [ 64.692393] dump_stack_lvl+0xa9/0x110 [ 64.694342] dump_stack+0x19/0x20 [ 64.695802] panic+0x4fd/0x520 [ 64.698074] ? watchdog_timer_fn+0x31e/0x390 [ 64.700692] watchdog_timer_fn+0x337/0x390 [ 64.702824] ? __pfx_watchdog_timer_fn+0x10/0x10 [ 64.704467] __hrtimer_run_queues+0x2ed/0x820 [ 64.707965] hrtimer_interrupt+0x134/0x2d0 [ 64.711106] __sysvec_apic_timer_interrupt+0x9c/0x290 [ 64.713132] sysvec_apic_timer_interrupt+0xab/0xe0 [ 64.714798] </IRQ> [ 64.715851] <TASK> [ 64.717322] asm_sysvec_apic_timer_interrupt+0x1f/0x30 [ 64.718915] RIP: 0010:__text_poke+0x2f2/0x4e0 [ 64.720373] Code: 06 00 20 00 00 e8 0e f0 07 00 48 81 7d b8 60 49 07 81 0f 84 79 01 00 00 48 83 7d 98 00 74 0b e8 24 f3 2c 00 fb 0f 1f 44 00 00 <48> 8b 7d c8 e8 85 dc fa 01 4c 89 f8 48 8b 4d d0 65 48 33 0c 25 28 [ 64.723885] RSP: 0018:ffffc90000f4fa20 EFLAGS: 00000206 [ 64.725754] RAX: 000000000009c7c3 RBX: 0000000000000000 RCX: 0000000000000006 [ 64.727543] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 [ 64.729319] RBP: ffffc90000f4fa90 R08: 0000000000000001 R09: 0000000000000001 [ 64.731110] R10: ffffc90000f4f790 R11: 0000000000000001 R12: ffff88800fe29980 [ 64.732904] R13: 0000000000000001 R14: 0000000000000da9 R15: ffffffff82263da5 [ 64.734725] ? drm_buddy_fini+0x5/0xe0 [ 64.739558] ? __pfx_text_poke_memcpy+0x10/0x10 [ 64.741484] ? drm_buddy_fini+0x5/0xe0 [ 64.743608] text_poke+0x3a/0x60 [ 64.745483] text_poke_bp_batch+0x172/0x310 [ 64.747621] ? drm_atomic_helper_page_flip_target+0x4/0x170 [ 64.749968] text_poke_queue+0x93/0xb0 [ 64.752071] ftrace_replace_code+0x12a/0x1b0 [ 64.754525] ftrace_modify_all_code+0x1b9/0x2a0 [ 64.756744] arch_ftrace_update_code+0xd/0x20 [ 64.758365] ftrace_startup_enable+0x67/0xa0 [ 64.760256] ftrace_startup+0x124/0x200 [ 64.762346] register_ftrace_function_nolock+0x43/0x90 [ 64.764363] register_ftrace_function+0x1eb/0x280 [ 64.766305] ? __sanitizer_cov_trace_switch+0x57/0xa0 [ 64.768721] perf_ftrace_event_register+0xcd/0xf0 [ 64.770672] perf_trace_event_init+0x98/0x4b0 [ 64.773005] perf_trace_init+0xde/0x170 [ 64.775114] perf_tp_event_init+0x60/0xa0 [ 64.776955] perf_try_init_event+0x88/0x280 [ 64.779378] perf_event_alloc+0xe25/0x1c00 [ 64.781369] ? perf_event_alloc+0x5/0x1c00 [ 64.784210] __do_sys_perf_event_open+0x3b6/0x1910 [ 64.790759] __x64_sys_perf_event_open+0x2f/0x40 [ 64.792600] do_syscall_64+0x3b/0x90 [ 64.794222] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 64.795798] RIP: 0033:0x7f768092b59d [ 64.797243] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d3 08 0d 00 f7 d8 64 89 01 48 [ 64.800765] RSP: 002b:00007fff176ab6a8 EFLAGS: 00000246 ORIG_RAX: 000000000000012a [ 64.803000] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f768092b59d [ 64.804790] RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 00000000200000c0 [ 64.806585] RBP: 00007fff176ab6c0 R08: 0000000000000000 R09: 00007fff176ab6c0 [ 64.808394] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000401180 [ 64.810181] R13: 00007fff176ab7e0 R14: 0000000000000000 R15: 0000000000000000 [ 64.815000] </TASK> [ 64.816463] Kernel Offset: disabled [ 64.818327] ---[ end Kernel panic - not syncing: softlockup: hung tasks ]--- " It could be reproduced in 1800s. I hope it's helpful. Thanks! --- If you don't need the following environment to reproduce the problem or if you already have one, please ignore the following information. How to reproduce: git clone https://gitlab.com/xupengfe/repro_vm_env.git cd repro_vm_env tar -xvf repro_vm_env.tar.gz cd repro_vm_env; ./start3.sh // it needs qemu-system-x86_64 and I used v7.1.0 // start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel // You could change the bzImage_xxx as you want You could use below command to log in, there is no password for root. ssh -p 10023 root@localhost After login vm(virtual machine) successfully, you could transfer reproduced binary to the vm by below way, and reproduce the problem in vm: gcc -pthread -o repro repro.c scp -P 10023 repro root@localhost:/root/ Get the bzImage for target kernel: Please use target kconfig and copy it to kernel_src/.config make olddefconfig make -jx bzImage //x should equal or less than cpu num your pc has Fill the bzImage file into above start3.sh to load the target kernel in vm. Tips: If you already have qemu-system-x86_64, please ignore below info. If you want to install qemu v7.1.0 version: git clone https://github.com/qemu/qemu.git cd qemu git checkout -f v7.1.0 mkdir build cd build yum install -y ninja-build.x86_64 ../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl make make install Thanks! BR.
