On 24/10/2024 22:13, Dave Chinner wrote:
BTW, can you please share logs which would contain full stacktraces that
this softlockup reports produce? The attached dmesg is just from fresh
boot... Thanks!
thanks for getting back to me.
So I think that enabling /proc/sys/kernel/softlockup_all_cpu_backtrace is
required there. Unfortunately my VM often just locks up without any sign of
life.
Attach a "serial" console to the vm - add "console=ttyS0,115600" to
the kernel command line and add "-serial pty" to the qemu command
line. You can then attach something like minicom to the /dev/pts/X
device that qemu creates for the console output and capture
everything from initial boot right through to the softlockup traces
that are emitted...
I am using an OCI instance, so I can't change the qemu command line (as
far as I know).
For this issue, the Cloud Shell locks up also. There are other console
connection methods, which I can try.
BTW, earlier today I got this once when trying to recreate this issue:
[ 1549.241972] ------------[ cut here ]------------
[ 1609.240236] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1609.240243] rcu: 5-...!: (0 ticks this GP)
idle=a8f4/1/0x4000000000000000 softirq=71287/71287 fqs=1
[ 1609.240249] rcu: (detected by 2, t=60004 jiffies, g=168077,
q=10823 ncpus=16)
[ 1609.240252] Sending NMI from CPU 2 to CPUs 5:
[ 1609.240277] NMI backtrace for cpu 5
[ 1609.240281] CPU: 5 UID: 1002 PID: 8250 Comm: mysqld Tainted: G
W 6.12.0-rc4-g556c97f2ecbf #40
[ 1609.240286] Tainted: [W]=WARN
[ 1609.240288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1609.240289] RIP: 0010:native_halt+0xe/0x20
[ 1609.240296] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1609.240298] RSP: 0018:ffffc0c8c71dbd20 EFLAGS: 00000046
[ 1609.240301] RAX: 0000000000000003 RBX: ffff9ff73fab6580 RCX:
0000000000000008
[ 1609.240303] RDX: ffff9ff7bffaf740 RSI: 0000000000000003 RDI:
ffff9ff73fab6580
[ 1609.240304] RBP: ffff9ff73f8b7440 R08: 0000000000000008 R09:
0000000000000074
[ 1609.240306] R10: 0000000000000002 R11: 0000000000000000 R12:
0000000000000000
[ 1609.240307] R13: 0000000000000001 R14: 0000000000000100 R15:
0000000000180000
[ 1609.240311] FS: 00007f9e12600700(0000) GS:ffff9ff73f880000(0000)
knlGS:0000000000000000
[ 1609.240313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1609.240315] CR2: 00007f9d63e00004 CR3: 0000001a0bc04005 CR4:
0000000000770ef0
[ 1609.240319] PKRU: 55555554
[ 1609.240320] Call Trace:
[ 1609.240322] <NMI>
[ 1609.240325] ? nmi_cpu_backtrace+0x98/0x110
[ 1609.240330] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1609.240334] ? nmi_handle+0x5c/0x150
[ 1609.240339] ? default_do_nmi+0x4e/0x120
[ 1609.240343] ? exc_nmi+0x137/0x1d0
[ 1609.240347] ? end_repeat_nmi+0xf/0x53
[ 1609.240354] ? native_halt+0xe/0x20
[ 1609.240357] ? native_halt+0xe/0x20
[ 1609.240360] ? native_halt+0xe/0x20
[ 1609.240363] </NMI>
[ 1609.240364] <TASK>
[ 1609.240366] kvm_wait+0x47/0x60
[ 1609.240368] __pv_queued_spin_lock_slowpath+0x255/0x370
[ 1609.240373] _raw_spin_lock+0x29/0x30
[ 1609.240376] raw_spin_rq_lock_nested+0x1c/0x80
[ 1609.240381] __task_rq_lock+0x3f/0xe0
[ 1609.240384] try_to_wake_up+0x3cf/0x640
[ 1609.240387] ? plist_del+0x63/0xc0
[ 1609.240391] wake_up_q+0x4d/0x90
[ 1609.240394] futex_wake+0x154/0x180
[ 1609.240400] do_futex+0xf8/0x1d0
[ 1609.240404] __x64_sys_futex+0x68/0x1c0
[ 1609.240407] ? restore_fpregs_from_fpstate+0x3c/0xa0
[ 1609.240411] do_syscall_64+0x62/0x170
[ 1609.240416] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1609.240419] RIP: 0033:0x7fa6a961191a
[ 1609.240422] Code: 00 00 b8 ca 00 00 00 0f 05 5a 5e c3 0f 1f 40 00 56
52 c7 07 00 00 00 00 81 f6 81 00 00 00 ba 01 00 00 00 b8 ca 00 00 00 0f
05 <5a> 5e c3 0f 1f 00 41 54 41 55 49 89 fc 49 89 f5 48 83 ec 18 48 89
[ 1609.240424] RSP: 002b:00007f9e125feee0 EFLAGS: 00000206 ORIG_RAX:
00000000000000ca
[ 1609.240426] RAX: ffffffffffffffda RBX: 00007fa6a02ccb50 RCX:
00007fa6a961191a
[ 1609.240428] RDX: 0000000000000001 RSI: 0000000000000081 RDI:
00007fa6a02ccb50
[ 1609.240429] RBP: 00007f9e125ff1a0 R08: 0000000000000000 R09:
00007fa6a9d0c000
[ 1609.240431] R10: 00007fa6a9d0b080 R11: 0000000000000206 R12:
00007fa6a9852830
[ 1609.240432] R13: 0f83e0f83e0f83e1 R14: 0000000000000008 R15:
0000000000000100
[ 1609.240437] </TASK>
[ 1609.241262] rcu: rcu_preempt kthread timer wakeup didn't happen for
59997 jiffies! g168077 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[ 1609.241265] rcu: Possible timer handling issue on cpu=5
timer-softirq=15698
[ 1609.241267] rcu: rcu_preempt kthread starved for 60000 jiffies!
g168077 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=5
[ 1609.241269] rcu: Unless rcu_preempt kthread gets sufficient CPU
time, OOM is now expected behavior.
[ 1609.241270] rcu: RCU grace-period kthread stack dump:
[ 1609.241271] task:rcu_preempt state:I stack:0 pid:17
tgid:17 ppid:2 flags:0x00004000
[ 1609.241275] Call Trace:
[ 1609.241277] <TASK>
[ 1609.241279] __schedule+0x334/0xba0
[ 1609.241286] schedule+0x36/0xc0
[ 1609.241290] schedule_timeout+0x1e0/0x2c0
[ 1609.241293] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1609.241296] ? __pfx_process_timeout+0x10/0x10
[ 1609.241300] rcu_gp_fqs_loop+0x336/0x500
[ 1609.241306] ? __pfx_rcu_gp_kthread+0x10/0x10
[ 1609.241308] rcu_gp_kthread+0xdf/0x190
[ 1609.241311] kthread+0xd2/0x100
[ 1609.241315] ? __pfx_kthread+0x10/0x10
[ 1609.241318] ret_from_fork+0x34/0x40
[ 1609.241321] ? __pfx_kthread+0x10/0x10
[ 1609.241324] ret_from_fork_asm+0x1a/0x30
[ 1609.241331] </TASK>
[ 1669.246372] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 1669.246379] rcu: 0-...0: (1 GPs behind) idle=dff4/0/0x1
softirq=83650/83651 fqs=6001
[ 1669.246385] rcu: 1-...0: (1 GPs behind) idle=eb2c/0/0x1
softirq=74528/74529 fqs=6001
[ 1669.246388] rcu: 4-...0: (1 GPs behind) idle=4b84/0/0x1
softirq=78302/78303 fqs=6001
[ 1669.246391] rcu: 5-...0: (1 GPs behind)
idle=a8f4/1/0x4000000000000000 softirq=71287/71287 fqs=6001
[ 1669.246394] rcu: 7-...0: (1 GPs behind)
idle=1a3c/1/0x4000000000000000 softirq=71163/71164 fqs=6001
[ 1669.246396] rcu: 8-...0: (1 GPs behind)
idle=a654/1/0x4000000000000000 softirq=78387/78388 fqs=6001
[ 1669.246398] rcu: 9-...0: (1 GPs behind)
idle=8b0c/1/0x4000000000000000 softirq=91122/91123 fqs=6001
[ 1669.246401] rcu: (detected by 13, t=60005 jiffies, g=168081,
q=14788 ncpus=16)
[ 1669.246404] Sending NMI from CPU 13 to CPUs 0:
[ 1669.246431] NMI backtrace for cpu 0
[ 1669.246439] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1669.246445] Tainted: [W]=WARN
[ 1669.246446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.246448] RIP: 0010:native_halt+0xe/0x20
[ 1669.246455] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1669.246458] RSP: 0018:ffffc0c8c0003d58 EFLAGS: 00000046
[ 1669.246461] RAX: 0000000000000001 RBX: ffff9ff73fab6580 RCX:
0000000000000001
[ 1669.246463] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff9ff73f637454
[ 1669.246464] RBP: ffff9ff73f637440 R08: 0000000000000001 R09:
0000000000000200
[ 1669.246466] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9ff73f6b7440
[ 1669.246467] R13: ffff9ff73f637454 R14: 0000000000000001 R15:
0000000000040000
[ 1669.246471] FS: 0000000000000000(0000) GS:ffff9ff73f600000(0000)
knlGS:0000000000000000
[ 1669.246474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.246476] CR2: 00007f8f73400004 CR3: 0000001a0bc04005 CR4:
0000000000770ef0
[ 1669.246479] PKRU: 55555554
[ 1669.246481] Call Trace:
[ 1669.246484] <NMI>
[ 1669.246487] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.246492] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.246496] ? nmi_handle+0x5c/0x150
[ 1669.246501] ? default_do_nmi+0x4e/0x120
[ 1669.246505] ? exc_nmi+0x137/0x1d0
[ 1669.246509] ? end_repeat_nmi+0xf/0x53
[ 1669.246516] ? native_halt+0xe/0x20
[ 1669.246520] ? native_halt+0xe/0x20
[ 1669.246523] ? native_halt+0xe/0x20
[ 1669.246526] </NMI>
[ 1669.246527] <IRQ>
[ 1669.246528] kvm_wait+0x47/0x60
[ 1669.246531] __pv_queued_spin_lock_slowpath+0x307/0x370
[ 1669.246536] _raw_spin_lock+0x29/0x30
[ 1669.246540] raw_spin_rq_lock_nested+0x1c/0x80
[ 1669.246544] _raw_spin_rq_lock_irqsave+0x17/0x20
[ 1669.246548] sched_balance_rq+0xa4e/0xd70
[ 1669.246557] sched_balance_domains+0x277/0x390
[ 1669.246561] _nohz_idle_balance.isra.146+0x2bb/0x3a0
[ 1669.246566] handle_softirqs+0xca/0x2c0
[ 1669.246572] irq_exit_rcu+0xb0/0xd0
[ 1669.246575] sysvec_call_function_single+0x71/0x90
[ 1669.246579] </IRQ>
[ 1669.246580] <TASK>
[ 1669.246581] asm_sysvec_call_function_single+0x1a/0x20
[ 1669.246585] RIP: 0010:pv_native_safe_halt+0xf/0x20
[ 1669.246587] Code: 22 d7 e9 6f 48 2d 00 0f 1f 40 00 90 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 43 76 3f 00 fb
f4 <e9> 47 48 2d 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[ 1669.246589] RSP: 0018:ffffffff95803e68 EFLAGS: 00000212
[ 1669.246591] RAX: 0000000000000000 RBX: ffffffff95810940 RCX:
0000000000000000
[ 1669.246593] RDX: 4000000000000000 RSI: 0000000000000087 RDI:
000000000129dfec
[ 1669.246594] RBP: 0000000000000000 R08: 0000000000000001 R09:
ffff9ff7bfffffc0
[ 1669.246596] R10: 000000000003c540 R11: 0000000000000000 R12:
ffffffff95db8be0
[ 1669.246597] R13: ffffffff95810940 R14: 0000000000000000 R15:
0000000000000000
[ 1669.246603] default_idle+0x9/0x20
[ 1669.246606] default_idle_call+0x34/0xf0
[ 1669.246608] do_idle+0x1f8/0x270
[ 1669.246613] cpu_startup_entry+0x29/0x30
[ 1669.246616] rest_init+0xcc/0xd0
[ 1669.246619] start_kernel+0x459/0x6a0
[ 1669.246626] x86_64_start_reservations+0x21/0x40
[ 1669.246631] x86_64_start_kernel+0x91/0xa0
[ 1669.246635] common_startup_64+0x13e/0x141
[ 1669.246641] </TASK>
[ 1669.247412] Sending NMI from CPU 13 to CPUs 1:
[ 1669.247431] NMI backtrace for cpu 1
[ 1669.247435] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1669.247438] Tainted: [W]=WARN
[ 1669.247439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.247440] RIP: 0010:native_halt+0xe/0x20
[ 1669.247442] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1669.247444] RSP: 0018:ffffc0c8c02dcd58 EFLAGS: 00000046
[ 1669.247446] RAX: 0000000000000001 RBX: ffff9ff73fab6580 RCX:
0000000000000001
[ 1669.247447] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff9ff73f6b7454
[ 1669.247448] RBP: ffff9ff73f6b7440 R08: 0000000000000001 R09:
0000000000000200
[ 1669.247449] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9ff73f8b7440
[ 1669.247450] R13: ffff9ff73f6b7454 R14: 0000000000000001 R15:
0000000000080000
[ 1669.247452] FS: 0000000000000000(0000) GS:ffff9ff73f680000(0000)
knlGS:0000000000000000
[ 1669.247454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.247455] CR2: 00007f9d22800004 CR3: 00000001095f2001 CR4:
0000000000770ef0
[ 1669.247457] PKRU: 55555554
[ 1669.247458] Call Trace:
[ 1669.247459] <NMI>
[ 1669.247460] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.247463] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.247465] ? nmi_handle+0x5c/0x150
[ 1669.247469] ? default_do_nmi+0x4e/0x120
[ 1669.247471] ? exc_nmi+0x137/0x1d0
[ 1669.247474] ? end_repeat_nmi+0xf/0x53
[ 1669.247479] ? native_halt+0xe/0x20
[ 1669.247482] ? native_halt+0xe/0x20
[ 1669.247485] ? native_halt+0xe/0x20
[ 1669.247487] </NMI>
[ 1669.247488] <IRQ>
[ 1669.247489] kvm_wait+0x47/0x60
[ 1669.247490] __pv_queued_spin_lock_slowpath+0x307/0x370
[ 1669.247493] _raw_spin_lock+0x29/0x30
[ 1669.247496] raw_spin_rq_lock_nested+0x1c/0x80
[ 1669.247498] _raw_spin_rq_lock_irqsave+0x17/0x20
[ 1669.247500] sched_balance_rq+0xa4e/0xd70
[ 1669.247507] sched_balance_domains+0x277/0x390
[ 1669.247511] _nohz_idle_balance.isra.146+0x2bb/0x3a0
[ 1669.247514] handle_softirqs+0xca/0x2c0
[ 1669.247518] irq_exit_rcu+0xb0/0xd0
[ 1669.247520] sysvec_call_function_single+0x71/0x90
[ 1669.247523] </IRQ>
[ 1669.247524] <TASK>
[ 1669.247525] asm_sysvec_call_function_single+0x1a/0x20
[ 1669.247528] RIP: 0010:pv_native_safe_halt+0xf/0x20
[ 1669.247529] Code: 22 d7 e9 6f 48 2d 00 0f 1f 40 00 90 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 43 76 3f 00 fb
f4 <e9> 47 48 2d 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[ 1669.247530] RSP: 0018:ffffc0c8c00c3ec8 EFLAGS: 00000202
[ 1669.247532] RAX: 0000000000000001 RBX: ffff9fd8809c8000 RCX:
0000000000000001
[ 1669.247533] RDX: 4000000000000000 RSI: 0000000000000087 RDI:
0000000000d4eb24
[ 1669.247534] RBP: 0000000000000001 R08: 0000000000000001 R09:
0101010101010101
[ 1669.247535] R10: 00000000ffffffff R11: 0000000000000001 R12:
ffffffff95db8be0
[ 1669.247536] R13: ffff9fd8809c8000 R14: 0000000000000000 R15:
0000000000000000
[ 1669.247541] default_idle+0x9/0x20
[ 1669.247543] default_idle_call+0x34/0xf0
[ 1669.247545] do_idle+0x1f8/0x270
[ 1669.247548] cpu_startup_entry+0x29/0x30
[ 1669.247550] start_secondary+0x11e/0x140
[ 1669.247554] common_startup_64+0x13e/0x141
[ 1669.247559] </TASK>
[ 1669.248416] Sending NMI from CPU 13 to CPUs 4:
[ 1669.248428] NMI backtrace for cpu 4
[ 1669.248431] CPU: 4 UID: 0 PID: 0 Comm: swapper/4 Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1669.248434] Tainted: [W]=WARN
[ 1669.248435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.248436] RIP: 0010:vprintk_emit+0x3a0/0x430
[ 1669.248440] Code: 6c 74 76 48 c7 c7 a8 0c 7f 96 c6 05 74 64 a8 02 01
e8 a4 32 cd 00 0f b6 05 68 64 a8 02 48 c7 c2 9b 0c 7f 96 84 c0 74 09 f3
90 <0f> b6 02 84 c0 75 f7 e8 04 1e 00 00 80 e7 02 74 06 fb 0f 1f 44 00
[ 1669.248442] RSP: 0018:ffffc0c8c01a4e18 EFLAGS: 00000002
[ 1669.248444] RAX: 0000000000000001 RBX: 0000000000000246 RCX:
0000000000000000
[ 1669.248445] RDX: ffffffff967f0c9b RSI: 0000000000000002 RDI:
ffffffff967f0ca8
[ 1669.248446] RBP: 0000000000000027 R08: 0000000000000000 R09:
c0000000ffff7fff
[ 1669.248448] R10: 0000000000000001 R11: ffffc0c8c01a4c48 R12:
0000000000000000
[ 1669.248449] R13: 0000000000000000 R14: ffffffff954639d9 R15:
ffffc0c8c01a4e70
[ 1669.248451] FS: 0000000000000000(0000) GS:ffff9ff73f800000(0000)
knlGS:0000000000000000
[ 1669.248453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.248454] CR2: 00007fa3de41be6c CR3: 0000000109516004 CR4:
0000000000770ef0
[ 1669.248457] PKRU: 55555554
[ 1669.248458] Call Trace:
[ 1669.248459] <NMI>
[ 1669.248461] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.248464] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.248466] ? nmi_handle+0x5c/0x150
[ 1669.248470] ? default_do_nmi+0x4e/0x120
[ 1669.248472] ? exc_nmi+0x137/0x1d0
[ 1669.248475] ? end_repeat_nmi+0xf/0x53
[ 1669.248480] ? vprintk_emit+0x3a0/0x430
[ 1669.248482] ? vprintk_emit+0x3a0/0x430
[ 1669.248485] ? vprintk_emit+0x3a0/0x430
[ 1669.248487] </NMI>
[ 1669.248488] <IRQ>
[ 1669.248489] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.248491] ? __pfx_workqueue_timedout+0x10/0x10
[ 1669.248494] ? __pfx_workqueue_timedout+0x10/0x10
[ 1669.248496] _printk+0x5c/0x80
[ 1669.248500] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.248502] workqueue_timedout+0x23/0x60
[ 1669.248505] call_timer_fn+0x27/0x130
[ 1669.248509] __run_timer_base.part.35+0x1fe/0x230
[ 1669.248512] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.248513] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.248515] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.248516] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.248518] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.248519] ? sched_clock_cpu+0x11/0x190
[ 1669.248524] run_timer_softirq+0x51/0x90
[ 1669.248526] handle_softirqs+0xca/0x2c0
[ 1669.248530] irq_exit_rcu+0xb0/0xd0
[ 1669.248532] sysvec_apic_timer_interrupt+0x71/0x90
[ 1669.248536] </IRQ>
[ 1669.248537] <TASK>
[ 1669.248538] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 1669.248540] RIP: 0010:pv_native_safe_halt+0xf/0x20
[ 1669.248542] Code: 22 d7 e9 6f 48 2d 00 0f 1f 40 00 90 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 43 76 3f 00 fb
f4 <e9> 47 48 2d 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
[ 1669.248544] RSP: 0018:ffffc0c8c00dbec8 EFLAGS: 00000216
[ 1669.248545] RAX: 0000000000000004 RBX: ffff9fd8809e0000 RCX:
ffff9fd884428e38
[ 1669.248546] RDX: 4000000000000000 RSI: 0000000000000004 RDI:
0000000000b34b7c
[ 1669.248547] RBP: 0000000000000004 R08: 0000000000000001 R09:
0000000000000004
[ 1669.248548] R10: 0000000000000000 R11: 0000000000000001 R12:
ffffffff95db8be0
[ 1669.248549] R13: ffff9fd8809e0000 R14: 0000000000000000 R15:
0000000000000000
[ 1669.248554] default_idle+0x9/0x20
[ 1669.248556] default_idle_call+0x34/0xf0
[ 1669.248558] do_idle+0x1f8/0x270
[ 1669.248561] cpu_startup_entry+0x29/0x30
[ 1669.248564] start_secondary+0x11e/0x140
[ 1669.248566] common_startup_64+0x13e/0x141
[ 1669.248571] </TASK>
[ 1669.249421] Sending NMI from CPU 13 to CPUs 5:
[ 1669.249444] NMI backtrace for cpu 5
[ 1669.249446] CPU: 5 UID: 1002 PID: 8250 Comm: mysqld Tainted: G
W 6.12.0-rc4-g556c97f2ecbf #40
[ 1669.249450] Tainted: [W]=WARN
[ 1669.249451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.249453] RIP: 0010:native_halt+0xe/0x20
[ 1669.249456] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1669.249458] RSP: 0018:ffffc0c8c71dbd20 EFLAGS: 00000046
[ 1669.249460] RAX: 0000000000000003 RBX: ffff9ff73fab6580 RCX:
0000000000000008
[ 1669.249462] RDX: 0000000000000000 RSI: 0000000000000003 RDI:
ffff9ff73fab6580
[ 1669.249463] RBP: ffff9ff73f8b7440 R08: 0000000000000008 R09:
0000000000000074
[ 1669.249464] R10: 0000000000000002 R11: 0000000000000000 R12:
0000000000000000
[ 1669.249466] R13: 0000000000000001 R14: 0000000000000100 R15:
0000000000180000
[ 1669.249468] FS: 00007f9e12600700(0000) GS:ffff9ff73f880000(0000)
knlGS:0000000000000000
[ 1669.249470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.249472] CR2: 00007f9d63e00004 CR3: 0000001a0bc04005 CR4:
0000000000770ef0
[ 1669.249475] PKRU: 55555554
[ 1669.249476] Call Trace:
[ 1669.249477] <NMI>
[ 1669.249479] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.249482] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.249484] ? nmi_handle+0x5c/0x150
[ 1669.249488] ? default_do_nmi+0x4e/0x120
[ 1669.249491] ? exc_nmi+0x137/0x1d0
[ 1669.249495] ? end_repeat_nmi+0xf/0x53
[ 1669.249501] ? native_halt+0xe/0x20
[ 1669.249504] ? native_halt+0xe/0x20
[ 1669.249507] ? native_halt+0xe/0x20
[ 1669.249511] </NMI>
[ 1669.249511] <TASK>
[ 1669.249512] kvm_wait+0x47/0x60
[ 1669.249515] __pv_queued_spin_lock_slowpath+0x255/0x370
[ 1669.249518] _raw_spin_lock+0x29/0x30
[ 1669.249521] raw_spin_rq_lock_nested+0x1c/0x80
[ 1669.249524] __task_rq_lock+0x3f/0xe0
[ 1669.249528] try_to_wake_up+0x3cf/0x640
[ 1669.249531] ? plist_del+0x63/0xc0
[ 1669.249534] wake_up_q+0x4d/0x90
[ 1669.249537] futex_wake+0x154/0x180
[ 1669.249544] do_futex+0xf8/0x1d0
[ 1669.249547] __x64_sys_futex+0x68/0x1c0
[ 1669.249550] ? restore_fpregs_from_fpstate+0x3c/0xa0
[ 1669.249554] do_syscall_64+0x62/0x170
[ 1669.249558] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1669.249561] RIP: 0033:0x7fa6a961191a
[ 1669.249564] Code: 00 00 b8 ca 00 00 00 0f 05 5a 5e c3 0f 1f 40 00 56
52 c7 07 00 00 00 00 81 f6 81 00 00 00 ba 01 00 00 00 b8 ca 00 00 00 0f
05 <5a> 5e c3 0f 1f 00 41 54 41 55 49 89 fc 49 89 f5 48 83 ec 18 48 89
[ 1669.249566] RSP: 002b:00007f9e125feee0 EFLAGS: 00000206 ORIG_RAX:
00000000000000ca
[ 1669.249568] RAX: ffffffffffffffda RBX: 00007fa6a02ccb50 RCX:
00007fa6a961191a
[ 1669.249570] RDX: 0000000000000001 RSI: 0000000000000081 RDI:
00007fa6a02ccb50
[ 1669.249571] RBP: 00007f9e125ff1a0 R08: 0000000000000000 R09:
00007fa6a9d0c000
[ 1669.249572] R10: 00007fa6a9d0b080 R11: 0000000000000206 R12:
00007fa6a9852830
[ 1669.249574] R13: 0f83e0f83e0f83e1 R14: 0000000000000008 R15:
0000000000000100
[ 1669.249579] </TASK>
[ 1669.250426] Sending NMI from CPU 13 to CPUs 7:
[ 1669.250443] NMI backtrace for cpu 7
[ 1669.250446] CPU: 7 UID: 0 PID: 9912 Comm: pidstat Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1669.250450] Tainted: [W]=WARN
[ 1669.250451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.250452] RIP: 0010:native_halt+0xe/0x20
[ 1669.250456] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1669.250458] RSP: 0018:ffffc0c8c8a4bb18 EFLAGS: 00000046
[ 1669.250460] RAX: 0000000000000003 RBX: ffff9ff73fab58c0 RCX:
0000000000000008
[ 1669.250461] RDX: ffff9ff7bffafb40 RSI: 0000000000000003 RDI:
ffff9ff73fab58c0
[ 1669.250462] RBP: ffff9ff73f9b7440 R08: 0000000000000008 R09:
00000000000000b4
[ 1669.250464] R10: ffffc0c8c8a4bbe0 R11: 0000000000000000 R12:
0000000000000000
[ 1669.250465] R13: 0000000000000001 R14: 0000000000000100 R15:
0000000000200000
[ 1669.250467] FS: 00007f461edcb580(0000) GS:ffff9ff73f980000(0000)
knlGS:0000000000000000
[ 1669.250469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.250470] CR2: 000055b6651fe000 CR3: 00000001082de002 CR4:
0000000000770ef0
[ 1669.250473] PKRU: 55555554
[ 1669.250474] Call Trace:
[ 1669.250475] <NMI>
[ 1669.250477] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.250481] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.250483] ? nmi_handle+0x5c/0x150
[ 1669.250486] ? default_do_nmi+0x4e/0x120
[ 1669.250489] ? exc_nmi+0x137/0x1d0
[ 1669.250492] ? end_repeat_nmi+0xf/0x53
[ 1669.250497] ? native_halt+0xe/0x20
[ 1669.250499] ? native_halt+0xe/0x20
[ 1669.250502] ? native_halt+0xe/0x20
[ 1669.250505] </NMI>
[ 1669.250506] <TASK>
[ 1669.250506] kvm_wait+0x47/0x60
[ 1669.250508] __pv_queued_spin_lock_slowpath+0x255/0x370
[ 1669.250512] _raw_spin_lock_irq+0x2f/0x40
[ 1669.250513] wq_worker_comm+0xbb/0x120
[ 1669.250517] proc_task_name+0xbb/0xf0
[ 1669.250522] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.250525] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.250527] do_task_stat+0x47f/0xec0
[ 1669.250535] proc_single_show+0x59/0xd0
[ 1669.250539] seq_read_iter+0x19f/0x410
[ 1669.250544] seq_read+0x109/0x150
[ 1669.250549] vfs_read+0xc2/0x350
[ 1669.250552] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.250554] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.250557] ksys_read+0x63/0xe0
[ 1669.250560] do_syscall_64+0x62/0x170
[ 1669.250563] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1669.250566] RIP: 0033:0x7f461e71fdf5
[ 1669.250568] Code: fe ff ff 50 48 8d 3d a2 8d 06 00 e8 35 ee 01 00 0f
1f 44 00 00 f3 0f 1e fa 48 8d 05 45 49 2a 00 8b 00 85 c0 75 0f 31 c0 0f
05 <48> 3d 00 f0 ff ff 77 53 c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89
[ 1669.250570] RSP: 002b:00007ffc727410d8 EFLAGS: 00000246 ORIG_RAX:
0000000000000000
[ 1669.250572] RAX: ffffffffffffffda RBX: 00007f461e3559b0 RCX:
00007f461e71fdf5
[ 1669.250573] RDX: 0000000000000400 RSI: 000055b64c60f5c0 RDI:
0000000000000004
[ 1669.250574] RBP: 0000000000000004 R08: 0000000000000000 R09:
0000000000000000
[ 1669.250575] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000000
[ 1669.250576] R13: 00000000000002a7 R14: 00007ffc727411f4 R15:
000055b64c60f130
[ 1669.250580] </TASK>
[ 1669.251430] Sending NMI from CPU 13 to CPUs 8:
[ 1669.251457] NMI backtrace for cpu 8
[ 1669.251461] CPU: 8 UID: 1002 PID: 8288 Comm: mysqld Tainted: G
W 6.12.0-rc4-g556c97f2ecbf #40
[ 1669.251467] Tainted: [W]=WARN
[ 1669.251468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.251470] RIP: 0010:native_halt+0xe/0x20
[ 1669.251478] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1669.251480] RSP: 0018:ffffc0c8c72cba20 EFLAGS: 00000046
[ 1669.251483] RAX: 0000000000000001 RBX: ffff9ff73fab6580 RCX:
0000000000000001
[ 1669.251485] RDX: 0000000000000001 RSI: 0000000000000001 RDI:
ffff9ff73fa37454
[ 1669.251487] RBP: ffff9ff73fa37440 R08: 0000000000000001 R09:
0000000000000200
[ 1669.251488] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9ff73f637440
[ 1669.251489] R13: ffff9ff73fa37454 R14: 0000000000000001 R15:
0000000000240000
[ 1669.251494] FS: 00007f9187200700(0000) GS:ffff9ff73fa00000(0000)
knlGS:0000000000000000
[ 1669.251496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.251498] CR2: 00007f8ff7800004 CR3: 0000001a0bc04002 CR4:
0000000000770ef0
[ 1669.251501] PKRU: 55555554
[ 1669.251502] Call Trace:
[ 1669.251505] <NMI>
[ 1669.251508] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.251514] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.251518] ? nmi_handle+0x5c/0x150
[ 1669.251523] ? default_do_nmi+0x4e/0x120
[ 1669.251527] ? exc_nmi+0x137/0x1d0
[ 1669.251531] ? end_repeat_nmi+0xf/0x53
[ 1669.251540] ? native_halt+0xe/0x20
[ 1669.251543] ? native_halt+0xe/0x20
[ 1669.251547] ? native_halt+0xe/0x20
[ 1669.251550] </NMI>
[ 1669.251551] <TASK>
[ 1669.251552] kvm_wait+0x47/0x60
[ 1669.251555] __pv_queued_spin_lock_slowpath+0x307/0x370
[ 1669.251560] _raw_spin_lock+0x29/0x30
[ 1669.251564] raw_spin_rq_lock_nested+0x1c/0x80
[ 1669.251568] _raw_spin_rq_lock_irqsave+0x17/0x20
[ 1669.251572] sched_balance_rq+0xa4e/0xd70
[ 1669.251581] sched_balance_newidle+0x1c3/0x430
[ 1669.251585] pick_next_task_fair+0x36/0x370
[ 1669.251588] ? dl_server_stop+0x2c/0x40
[ 1669.251592] pick_next_task+0x6b/0xc00
[ 1669.251595] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.251597] ? dequeue_task_fair+0xa1/0x2b0
[ 1669.251601] __schedule+0x155/0xba0
[ 1669.251606] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.251609] schedule+0x36/0xc0
[ 1669.251613] futex_wait_queue+0x66/0xa0
[ 1669.251619] __futex_wait+0x13d/0x1b0
[ 1669.251623] ? __pfx_futex_wake_mark+0x10/0x10
[ 1669.251629] futex_wait+0x6e/0x110
[ 1669.251632] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.251634] ? futex_wake+0x85/0x180
[ 1669.251639] do_futex+0x11a/0x1d0
[ 1669.251642] __x64_sys_futex+0x68/0x1c0
[ 1669.251645] ? restore_fpregs_from_fpstate+0x3c/0xa0
[ 1669.251650] do_syscall_64+0x62/0x170
[ 1669.251655] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1669.251658] RIP: 0033:0x7fa6a960e4ac
[ 1669.251662] Code: 68 4c 89 64 24 70 89 6c 24 78 e8 ef 2e 00 00 e8 da
32 00 00 45 31 d2 31 d2 8b 74 24 34 41 89 c0 4c 89 ff b8 ca 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 54 44 89 c7 e8 14 33 00 00 48 8b 7c 24 10 31
[ 1669.251664] RSP: 002b:00007f91871ff9f0 EFLAGS: 00000246 ORIG_RAX:
00000000000000ca
[ 1669.251667] RAX: ffffffffffffffda RBX: 00007fa6a001d518 RCX:
00007fa6a960e4ac
[ 1669.251668] RDX: 0000000000000000 RSI: 0000000000000080 RDI:
00007fa6a001d540
[ 1669.251669] RBP: 0000000000000000 R08: 0000000000000000 R09:
00007fa6a9d0c000
[ 1669.251671] R10: 0000000000000000 R11: 0000000000000246 R12:
00007fa6a001d4f0
[ 1669.251672] R13: 0000000000001eec R14: 0000000000000000 R15:
00007fa6a001d540
[ 1669.251677] </TASK>
[ 1669.252438] Sending NMI from CPU 13 to CPUs 9:
[ 1669.252457] NMI backtrace for cpu 9
[ 1669.252459] CPU: 9 UID: 1002 PID: 8255 Comm: mysqld Tainted: G
W 6.12.0-rc4-g556c97f2ecbf #40
[ 1669.252462] Tainted: [W]=WARN
[ 1669.252462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.252463] RIP: 0010:native_halt+0xe/0x20
[ 1669.252466] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1669.252468] RSP: 0018:ffffc0c8c7203718 EFLAGS: 00000046
[ 1669.252469] RAX: 0000000000000001 RBX: ffff9ff73fab6580 RCX:
0000000000000001
[ 1669.252470] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
ffff9ff73fab7454
[ 1669.252471] RBP: ffff9ff73fab7440 R08: ffff9ff73fab58e8 R09:
ffff9fd880401ff0
[ 1669.252473] R10: 0000000000000000 R11: ffffffff95869448 R12:
ffff9ff73fa37440
[ 1669.252474] R13: ffff9ff73fab7454 R14: 0000000000000001 R15:
0000000000280000
[ 1669.252476] FS: 00007f9e0a600700(0000) GS:ffff9ff73fa80000(0000)
knlGS:0000000000000000
[ 1669.252478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.252479] CR2: 00007f8fb5e00004 CR3: 0000001a0bc04004 CR4:
0000000000770ef0
[ 1669.252481] PKRU: 55555554
[ 1669.252482] Call Trace:
[ 1669.252483] <NMI>
[ 1669.252484] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.252487] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.252489] ? nmi_handle+0x5c/0x150
[ 1669.252493] ? default_do_nmi+0x4e/0x120
[ 1669.252495] ? exc_nmi+0x137/0x1d0
[ 1669.252498] ? end_repeat_nmi+0xf/0x53
[ 1669.252503] ? native_halt+0xe/0x20
[ 1669.252506] ? native_halt+0xe/0x20
[ 1669.252509] ? native_halt+0xe/0x20
[ 1669.252511] </NMI>
[ 1669.252512] <TASK>
[ 1669.252513] kvm_wait+0x47/0x60
[ 1669.252514] __pv_queued_spin_lock_slowpath+0x307/0x370
[ 1669.252518] _raw_spin_lock+0x29/0x30
[ 1669.252520] raw_spin_rq_lock_nested+0x1c/0x80
[ 1669.252523] try_to_wake_up+0x187/0x640
[ 1669.252526] kick_pool+0x65/0x140
[ 1669.252531] __queue_work+0x1ae/0x400
[ 1669.252535] queue_work_on+0x66/0x70
[ 1669.252538] soft_cursor+0x198/0x230
[ 1669.252542] bit_cursor+0x355/0x610
[ 1669.252546] ? __entry_text_end+0x101ec6/0x101ec9
[ 1669.252548] ? get_color+0x26/0x120
[ 1669.252552] ? __pfx_bit_cursor+0x10/0x10
[ 1669.252553] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.252556] hide_cursor+0x27/0x90
[ 1669.252559] vt_console_print+0x3e9/0x400
[ 1669.252564] console_flush_all+0x2df/0x500
[ 1669.252569] console_unlock+0x10b/0x1f0
[ 1669.252572] vprintk_emit+0x3c6/0x430
[ 1669.252575] _printk+0x5c/0x80
[ 1669.252580] __warn_printk+0xe3/0x120
[ 1669.252583] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.252587] pick_task_fair+0x103/0x160
[ 1669.252590] pick_next_task_fair+0x4c/0x370
[ 1669.252593] pick_next_task+0x6b/0xc00
[ 1669.252595] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.252597] ? dequeue_task_fair+0x4b/0x2b0
[ 1669.252599] __schedule+0x155/0xba0
[ 1669.252603] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.252605] schedule+0x36/0xc0
[ 1669.252608] futex_wait_queue+0x66/0xa0
[ 1669.252611] __futex_wait+0x13d/0x1b0
[ 1669.252615] ? __pfx_futex_wake_mark+0x10/0x10
[ 1669.252619] futex_wait+0x6e/0x110
[ 1669.252622] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.252623] ? futex_wake+0x85/0x180
[ 1669.252627] do_futex+0x11a/0x1d0
[ 1669.252630] __x64_sys_futex+0x68/0x1c0
[ 1669.252632] ? restore_fpregs_from_fpstate+0x3c/0xa0
[ 1669.252635] do_syscall_64+0x62/0x170
[ 1669.252638] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1669.252641] RIP: 0033:0x7fa6a961187d
[ 1669.252643] Code: f0 5a 41 5a c3 0f 1f 84 00 00 00 00 00 41 52 52 4d
31 d2 ba 02 00 00 00 81 f6 80 00 00 00 39 d0 75 08 90 b8 ca 00 00 00 0f
05 <89> d0 87 07 85 c0 75 f0 5a 41 5a c3 0f 1f 80 00 00 00 00 48 83 3a
[ 1669.252644] RSP: 002b:00007f9e0a5fee88 EFLAGS: 00000202 ORIG_RAX:
00000000000000ca
[ 1669.252646] RAX: ffffffffffffffda RBX: 00007fa6a02ccb50 RCX:
00007fa6a961187d
[ 1669.252647] RDX: 0000000000000002 RSI: 0000000000000080 RDI:
00007fa6a02ccb50
[ 1669.252648] RBP: 00007f9e0a5fef70 R08: 00007fa6a02ccb50 R09:
0000000000000028
[ 1669.252649] R10: 0000000000000000 R11: 0000000000000202 R12:
00007fa6a988c010
[ 1669.252650] R13: 0f83e0f83e0f83e1 R14: 0000000000000004 R15:
00007fa6a984a010
[ 1669.252654] </TASK>
[ 1669.253446] rcu: rcu_preempt kthread starved for 30005 jiffies!
g168081 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=2
[ 1669.253449] rcu: Unless rcu_preempt kthread gets sufficient CPU
time, OOM is now expected behavior.
[ 1669.253451] rcu: RCU grace-period kthread stack dump:
[ 1669.253452] task:rcu_preempt state:R running task stack:0
pid:17 tgid:17 ppid:2 flags:0x00004008
[ 1669.253457] Call Trace:
[ 1669.253458] <TASK>
[ 1669.253461] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.253464] ? sysvec_call_function_single+0xe/0x90
[ 1669.253467] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1669.253469] ? asm_sysvec_call_function_single+0x1a/0x20
[ 1669.253473] ? kvm_wait+0x47/0x60
[ 1669.253475] ? __pv_queued_spin_lock_slowpath+0x307/0x370
[ 1669.253478] ? _raw_spin_lock+0x29/0x30
[ 1669.253481] ? raw_spin_rq_lock_nested+0x1c/0x80
[ 1669.253484] ? resched_cpu+0x3e/0x80
[ 1669.253486] ? force_qs_rnp+0x239/0x2c0
[ 1669.253490] ? __pfx_rcu_watching_snap_recheck+0x10/0x10
[ 1669.253493] ? rcu_gp_fqs_loop+0x384/0x500
[ 1669.253497] ? __pfx_rcu_gp_kthread+0x10/0x10
[ 1669.253499] ? rcu_gp_kthread+0xdf/0x190
[ 1669.253502] ? kthread+0xd2/0x100
[ 1669.253506] ? __pfx_kthread+0x10/0x10
[ 1669.253509] ? ret_from_fork+0x34/0x40
[ 1669.253512] ? __pfx_kthread+0x10/0x10
[ 1669.253515] ? ret_from_fork_asm+0x1a/0x30
[ 1669.253521] </TASK>
[ 1669.253522] rcu: Stack dump where RCU GP kthread last ran:
[ 1669.253523] Sending NMI from CPU 13 to CPUs 2:
[ 1669.253539] NMI backtrace for cpu 2
[ 1669.253542] CPU: 2 UID: 0 PID: 17 Comm: rcu_preempt Tainted: G
W 6.12.0-rc4-g556c97f2ecbf #40
[ 1669.253546] Tainted: [W]=WARN
[ 1669.253547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1669.253548] RIP: 0010:native_halt+0xe/0x20
[ 1669.253553] Code: 0f 1f 84 00 00 00 00 00 0f 1f 40 00 90 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 23 f1 17 01
f4 <e9> 28 c3 05 01 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90
[ 1669.253555] RSP: 0018:ffffc0c8c009bd88 EFLAGS: 00000046
[ 1669.253557] RAX: 0000000000000001 RBX: ffff9ff73fab6580 RCX:
0000000000000001
[ 1669.253559] RDX: 0000000000000001 RSI: 0000000000000001 RDI:
ffff9ff73f737454
[ 1669.253560] RBP: ffff9ff73f737440 R08: ffff9ff73f7370f0 R09:
0000000000000377
[ 1669.253562] R10: 0000000000000001 R11: 0000000000000001 R12:
ffff9ff73fab7440
[ 1669.253563] R13: ffff9ff73f737454 R14: 0000000000000001 R15:
00000000000c0000
[ 1669.253566] FS: 0000000000000000(0000) GS:ffff9ff73f700000(0000)
knlGS:0000000000000000
[ 1669.253567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1669.253569] CR2: 00007f6d7d6f9890 CR3: 0000001a0bc04005 CR4:
0000000000770ef0
[ 1669.253572] PKRU: 55555554
[ 1669.253572] Call Trace:
[ 1669.253574] <NMI>
[ 1669.253575] ? nmi_cpu_backtrace+0x98/0x110
[ 1669.253579] ? nmi_cpu_backtrace_handler+0x11/0x20
[ 1669.253581] ? nmi_handle+0x5c/0x150
[ 1669.253585] ? default_do_nmi+0x4e/0x120
[ 1669.253587] ? exc_nmi+0x137/0x1d0
[ 1669.253590] ? end_repeat_nmi+0xf/0x53
[ 1669.253595] ? native_halt+0xe/0x20
[ 1669.253598] ? native_halt+0xe/0x20
[ 1669.253601] ? native_halt+0xe/0x20
[ 1669.253603] </NMI>
[ 1669.253604] <TASK>
[ 1669.253605] kvm_wait+0x47/0x60
[ 1669.253607] __pv_queued_spin_lock_slowpath+0x307/0x370
[ 1669.253610] _raw_spin_lock+0x29/0x30
[ 1669.253612] raw_spin_rq_lock_nested+0x1c/0x80
[ 1669.253615] resched_cpu+0x3e/0x80
[ 1669.253618] force_qs_rnp+0x239/0x2c0
[ 1669.253621] ? __pfx_rcu_watching_snap_recheck+0x10/0x10
[ 1669.253624] rcu_gp_fqs_loop+0x384/0x500
[ 1669.253628] ? __pfx_rcu_gp_kthread+0x10/0x10
[ 1669.253631] rcu_gp_kthread+0xdf/0x190
[ 1669.253633] kthread+0xd2/0x100
[ 1669.253637] ? __pfx_kthread+0x10/0x10
[ 1669.253640] ret_from_fork+0x34/0x40
[ 1669.253643] ? __pfx_kthread+0x10/0x10
[ 1669.253646] ret_from_fork_asm+0x1a/0x30
[ 1669.253652] </TASK>
[ 1721.537374] INFO: task kworker/u66:13:128 blocked for more than 122
seconds.
[ 1721.537380] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.537382] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.537383] task:kworker/u66:13 state:D stack:0 pid:128
tgid:128 ppid:2 flags:0x00004000
[ 1721.537388] Workqueue: events_unbound idle_cull_fn
[ 1721.537395] Call Trace:
[ 1721.537397] <TASK>
[ 1721.537401] __schedule+0x334/0xba0
[ 1721.537409] schedule+0x36/0xc0
[ 1721.537412] schedule_preempt_disabled+0x15/0x30
[ 1721.537415] __mutex_lock.isra.14+0x431/0x690
[ 1721.537418] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537421] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537424] idle_cull_fn+0x3b/0xe0
[ 1721.537428] process_scheduled_works+0x109/0x4e0
[ 1721.537431] ? __pfx_workqueue_timedout+0x10/0x10
[ 1721.537434] ? __pfx_idle_cull_fn+0x10/0x10
[ 1721.537437] worker_thread+0x117/0x240
[ 1721.537440] ? __pfx_worker_thread+0x10/0x10
[ 1721.537442] kthread+0xd2/0x100
[ 1721.537446] ? __pfx_kthread+0x10/0x10
[ 1721.537449] ret_from_fork+0x34/0x40
[ 1721.537453] ? __pfx_kthread+0x10/0x10
[ 1721.537456] ret_from_fork_asm+0x1a/0x30
[ 1721.537462] </TASK>
[ 1721.537469] INFO: task kworker/u65:4:229 blocked for more than 122
seconds.
[ 1721.537471] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.537472] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.537473] task:kworker/u65:4 state:D stack:0 pid:229
tgid:229 ppid:2 flags:0x00004000
[ 1721.537477] Workqueue: events_unbound idle_cull_fn
[ 1721.537479] Call Trace:
[ 1721.537481] <TASK>
[ 1721.537483] __schedule+0x334/0xba0
[ 1721.537488] schedule+0x36/0xc0
[ 1721.537491] schedule_preempt_disabled+0x15/0x30
[ 1721.537493] __mutex_lock.isra.14+0x431/0x690
[ 1721.537496] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537498] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537501] idle_cull_fn+0x3b/0xe0
[ 1721.537504] process_scheduled_works+0x109/0x4e0
[ 1721.537507] ? __pfx_workqueue_timedout+0x10/0x10
[ 1721.537509] ? __pfx_idle_cull_fn+0x10/0x10
[ 1721.537512] worker_thread+0x117/0x240
[ 1721.537515] ? __pfx_worker_thread+0x10/0x10
[ 1721.537517] kthread+0xd2/0x100
[ 1721.537520] ? __pfx_kthread+0x10/0x10
[ 1721.537523] ret_from_fork+0x34/0x40
[ 1721.537525] ? __pfx_kthread+0x10/0x10
[ 1721.537528] ret_from_fork_asm+0x1a/0x30
[ 1721.537533] </TASK>
[ 1721.537538] INFO: task kworker/11:1H:659 blocked for more than 122
seconds.
[ 1721.537540] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.537541] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.537542] task:kworker/11:1H state:D stack:0 pid:659
tgid:659 ppid:2 flags:0x00004000
[ 1721.537546] Workqueue: kblockd blk_mq_timeout_work
[ 1721.537550] Call Trace:
[ 1721.537552] <TASK>
[ 1721.537554] __schedule+0x334/0xba0
[ 1721.537559] schedule+0x36/0xc0
[ 1721.537562] schedule_timeout+0x283/0x2c0
[ 1721.537565] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537567] ? scsi_queue_rq+0x2b6/0xbe0
[ 1721.537571] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537573] ? __prepare_to_swait+0x52/0x80
[ 1721.537577] wait_for_completion_state+0x173/0x1d0
[ 1721.537581] __wait_rcu_gp+0x121/0x150
[ 1721.537585] synchronize_rcu_normal.part.63+0x3a/0x60
[ 1721.537589] ? __pfx_call_rcu_hurry+0x10/0x10
[ 1721.537592] ? __pfx_wakeme_after_rcu+0x10/0x10
[ 1721.537594] synchronize_rcu_normal+0x9a/0xb0
[ 1721.537597] ? __pfx_blk_mq_check_expired+0x10/0x10
[ 1721.537601] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537603] blk_mq_timeout_work+0x142/0x1a0
[ 1721.537605] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.537608] process_scheduled_works+0x109/0x4e0
[ 1721.537611] ? __pfx_workqueue_timedout+0x10/0x10
[ 1721.537614] ? __pfx_blk_mq_timeout_work+0x10/0x10
[ 1721.537617] worker_thread+0x117/0x240
[ 1721.537619] ? __pfx_worker_thread+0x10/0x10
[ 1721.537622] kthread+0xd2/0x100
[ 1721.537624] ? __pfx_kthread+0x10/0x10
[ 1721.537627] ret_from_fork+0x34/0x40
[ 1721.537630] ? __pfx_kthread+0x10/0x10
[ 1721.537633] ret_from_fork_asm+0x1a/0x30
[ 1721.537638] </TASK>
[ 1721.537697] INFO: task mysqld:8329 blocked for more than 122 seconds.
[ 1721.537699] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.537700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.537701] task:mysqld state:D stack:0 pid:8329
tgid:8190 ppid:8021 flags:0x00000000
[ 1721.537704] Call Trace:
[ 1721.537705] <TASK>
[ 1721.537707] __schedule+0x334/0xba0
[ 1721.537712] schedule+0x36/0xc0
[ 1721.537716] inode_dio_wait+0x7b/0xb0
[ 1721.537720] ? __pfx_var_wake_function+0x10/0x10
[ 1721.537726] xfs_file_fallocate+0xca/0x440 [xfs]
[ 1721.537873] vfs_fallocate+0x124/0x310
[ 1721.537878] ksys_fallocate+0x40/0x80
[ 1721.537881] __x64_sys_fallocate+0x1e/0x30
[ 1721.537884] do_syscall_64+0x62/0x170
[ 1721.537889] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1721.537892] RIP: 0033:0x7fa6a6d2499a
[ 1721.537895] RSP: 002b:00007fa6a43f5bd0 EFLAGS: 00000246 ORIG_RAX:
000000000000011d
[ 1721.537898] RAX: ffffffffffffffda RBX: 0000000004000000 RCX:
00007fa6a6d2499a
[ 1721.537899] RDX: 0000000250000000 RSI: 0000000000000000 RDI:
0000000000000029
[ 1721.537901] RBP: 00007fa6a43f6120 R08: 0000000000001000 R09:
0000000250000000
[ 1721.537902] R10: 0000000004000000 R11: 0000000000000246 R12:
00007fa6a02c1e10
[ 1721.537904] R13: 00007fa6a02c1ed8 R14: 00007f8bd8012f70 R15:
00007f8bd809a0d0
[ 1721.537908] </TASK>
[ 1721.537910] INFO: task mysqld:8331 blocked for more than 122 seconds.
[ 1721.537911] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.537913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.537914] task:mysqld state:D stack:0 pid:8331
tgid:8190 ppid:8021 flags:0x00000000
[ 1721.537916] Call Trace:
[ 1721.537926] <TASK>
[ 1721.537928] __schedule+0x334/0xba0
[ 1721.537933] schedule+0x36/0xc0
[ 1721.537937] inode_dio_wait+0x7b/0xb0
[ 1721.537940] ? __pfx_var_wake_function+0x10/0x10
[ 1721.537943] xfs_file_fallocate+0xca/0x440 [xfs]
[ 1721.538078] vfs_fallocate+0x124/0x310
[ 1721.538081] ksys_fallocate+0x40/0x80
[ 1721.538085] __x64_sys_fallocate+0x1e/0x30
[ 1721.538087] do_syscall_64+0x62/0x170
[ 1721.538091] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1721.538093] RIP: 0033:0x7fa6a6d2499a
[ 1721.538095] RSP: 002b:00007fa6a41f1bd0 EFLAGS: 00000246 ORIG_RAX:
000000000000011d
[ 1721.538097] RAX: ffffffffffffffda RBX: 0000000004000000 RCX:
00007fa6a6d2499a
[ 1721.538099] RDX: 000000024c000000 RSI: 0000000000000000 RDI:
0000000000000027
[ 1721.538100] RBP: 00007fa6a41f2120 R08: 0000000000001000 R09:
000000024c000000
[ 1721.538101] R10: 0000000004000000 R11: 0000000000000246 R12:
00007fa6a02c1fe0
[ 1721.538103] R13: 00007fa6a02c20a8 R14: 00007f8bcc012f80 R15:
00007f8bcc099f80
[ 1721.538107] </TASK>
[ 1721.538108] INFO: task mysqld:8332 blocked for more than 122 seconds.
[ 1721.538110] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.538111] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.538112] task:mysqld state:D stack:0 pid:8332
tgid:8190 ppid:8021 flags:0x00000000
[ 1721.538115] Call Trace:
[ 1721.538116] <TASK>
[ 1721.538118] __schedule+0x334/0xba0
[ 1721.538122] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.538125] schedule+0x36/0xc0
[ 1721.538129] schedule_timeout+0x1e0/0x2c0
[ 1721.538132] ? __pfx_process_timeout+0x10/0x10
[ 1721.538137] io_schedule_timeout+0x51/0x70
[ 1721.538140] __iomap_dio_rw+0x5ca/0x840
[ 1721.538145] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.538152] iomap_dio_rw+0x12/0x40
[ 1721.538155] xfs_file_dio_write_aligned+0xa6/0x130 [xfs]
[ 1721.538283] xfs_file_write_iter+0xcc/0x120 [xfs]
[ 1721.538427] vfs_write+0x2fc/0x430
[ 1721.538433] ksys_pwrite64+0x69/0xa0
[ 1721.538436] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.538439] do_syscall_64+0x62/0x170
[ 1721.538442] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1721.538445] RIP: 0033:0x7fa6a96124d7
[ 1721.538447] RSP: 002b:00007fa69fff65a0 EFLAGS: 00000293 ORIG_RAX:
0000000000000012
[ 1721.538449] RAX: ffffffffffffffda RBX: 000000000000002b RCX:
00007fa6a96124d7
[ 1721.538450] RDX: 0000000000100000 RSI: 00007f8bc491c000 RDI:
000000000000002b
[ 1721.538452] RBP: 00007f8bc491c000 R08: 0000000000000000 R09:
00007fa69fff88ec
[ 1721.538453] R10: 0000000248400000 R11: 0000000000000293 R12:
0000000000100000
[ 1721.538454] R13: 0000000248400000 R14: 00000000042ae5f8 R15:
00007f8bc409a270
[ 1721.538459] </TASK>
[ 1721.538460] INFO: task mysqld:8333 blocked for more than 122 seconds.
[ 1721.538462] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.538463] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.538464] task:mysqld state:D stack:0 pid:8333
tgid:8190 ppid:8021 flags:0x00000000
[ 1721.538467] Call Trace:
[ 1721.538468] <TASK>
[ 1721.538470] __schedule+0x334/0xba0
[ 1721.538475] schedule+0x36/0xc0
[ 1721.538478] inode_dio_wait+0x7b/0xb0
[ 1721.538482] ? __pfx_var_wake_function+0x10/0x10
[ 1721.538485] xfs_file_fallocate+0xca/0x440 [xfs]
[ 1721.538610] vfs_fallocate+0x124/0x310
[ 1721.538613] ksys_fallocate+0x40/0x80
[ 1721.538616] __x64_sys_fallocate+0x1e/0x30
[ 1721.538618] do_syscall_64+0x62/0x170
[ 1721.538622] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1721.538625] RIP: 0033:0x7fa6a6d2499a
[ 1721.538626] RSP: 002b:00007fa69f9f8bd0 EFLAGS: 00000246 ORIG_RAX:
000000000000011d
[ 1721.538628] RAX: ffffffffffffffda RBX: 0000000004000000 RCX:
00007fa6a6d2499a
[ 1721.538630] RDX: 0000000250000000 RSI: 0000000000000000 RDI:
0000000000000028
[ 1721.538631] RBP: 00007fa69f9f9120 R08: 0000000000001000 R09:
0000000250000000
[ 1721.538632] R10: 0000000004000000 R11: 0000000000000246 R12:
00007fa6a02c1bf0
[ 1721.538634] R13: 00007fa6a02c1cb8 R14: 00007f8bc0076090 R15:
00007f8bc009a880
[ 1721.538638] </TASK>
[ 1721.538639] INFO: task mysqld:8335 blocked for more than 122 seconds.
[ 1721.538641] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.538642] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.538643] task:mysqld state:D stack:0 pid:8335
tgid:8190 ppid:8021 flags:0x00000000
[ 1721.538645] Call Trace:
[ 1721.538647] <TASK>
[ 1721.538649] __schedule+0x334/0xba0
[ 1721.538654] schedule+0x36/0xc0
[ 1721.538657] inode_dio_wait+0x7b/0xb0
[ 1721.538660] ? __pfx_var_wake_function+0x10/0x10
[ 1721.538664] xfs_file_fallocate+0xca/0x440 [xfs]
[ 1721.538788] vfs_fallocate+0x124/0x310
[ 1721.538792] ksys_fallocate+0x40/0x80
[ 1721.538795] __x64_sys_fallocate+0x1e/0x30
[ 1721.538797] do_syscall_64+0x62/0x170
[ 1721.538801] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1721.538803] RIP: 0033:0x7fa6a6d2499a
[ 1721.538805] RSP: 002b:00007fa69e9f8bd0 EFLAGS: 00000246 ORIG_RAX:
000000000000011d
[ 1721.538807] RAX: ffffffffffffffda RBX: 0000000004000000 RCX:
00007fa6a6d2499a
[ 1721.538808] RDX: 0000000250000000 RSI: 0000000000000000 RDI:
0000000000000026
[ 1721.538810] RBP: 00007fa69e9f9120 R08: 0000000000001000 R09:
0000000250000000
[ 1721.538811] R10: 0000000004000000 R11: 0000000000000246 R12:
00007fa6a02c21b0
[ 1721.538812] R13: 00007fa6a02c2278 R14: 00007f8bb4056cb0 R15:
00007f8bb4099de0
[ 1721.538817] </TASK>
[ 1721.538819] INFO: task mysqld:8341 blocked for more than 122 seconds.
[ 1721.538820] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.538821] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.538822] task:mysqld state:D stack:0 pid:8341
tgid:8190 ppid:8021 flags:0x00000000
[ 1721.538825] Call Trace:
[ 1721.538827] <TASK>
[ 1721.538829] __schedule+0x334/0xba0
[ 1721.538833] schedule+0x36/0xc0
[ 1721.538837] inode_dio_wait+0x7b/0xb0
[ 1721.538840] ? __pfx_var_wake_function+0x10/0x10
[ 1721.538843] xfs_file_fallocate+0xca/0x440 [xfs]
[ 1721.538974] vfs_fallocate+0x124/0x310
[ 1721.538977] ksys_fallocate+0x40/0x80
[ 1721.538980] __x64_sys_fallocate+0x1e/0x30
[ 1721.538983] do_syscall_64+0x62/0x170
[ 1721.538986] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1721.538989] RIP: 0033:0x7fa6a6d2499a
[ 1721.538991] RSP: 002b:00007fa697ff8bd0 EFLAGS: 00000246 ORIG_RAX:
000000000000011d
[ 1721.538993] RAX: ffffffffffffffda RBX: 0000000004000000 RCX:
00007fa6a6d2499a
[ 1721.538994] RDX: 000000024c000000 RSI: 0000000000000000 RDI:
000000000000002c
[ 1721.538995] RBP: 00007fa697ff9120 R08: 0000000000001000 R09:
000000024c000000
[ 1721.538997] R10: 0000000004000000 R11: 0000000000000246 R12:
00007fa6a02c1850
[ 1721.538998] R13: 00007fa6a02c1918 R14: 00007f8b9001a950 R15:
00007f8b9003b5b0
[ 1721.539002] </TASK>
[ 1721.539006] INFO: task top:9931 blocked for more than 122 seconds.
[ 1721.539008] Tainted: G W
6.12.0-rc4-g556c97f2ecbf #40
[ 1721.539009] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1721.539010] task:top state:D stack:0 pid:9931
tgid:9931 ppid:9915 flags:0x00004002
[ 1721.539013] Call Trace:
[ 1721.539015] <TASK>
[ 1721.539017] __schedule+0x334/0xba0
[ 1721.539022] schedule+0x36/0xc0
[ 1721.539025] schedule_preempt_disabled+0x15/0x30
[ 1721.539027] __mutex_lock.isra.14+0x431/0x690
[ 1721.539029] ? prep_new_page+0x1b/0x50
[ 1721.539033] ? get_page_from_freelist+0x11b2/0x22a0
[ 1721.539037] wq_worker_comm+0x26/0x120
[ 1721.539041] proc_task_name+0xbb/0xf0
[ 1721.539045] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.539047] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.539050] do_task_stat+0x47f/0xec0
[ 1721.539058] proc_single_show+0x59/0xd0
[ 1721.539063] seq_read_iter+0x19f/0x410
[ 1721.539068] seq_read+0x109/0x150
[ 1721.539073] vfs_read+0xc2/0x350
[ 1721.539075] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.539077] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1721.539081] ksys_read+0x63/0xe0
[ 1721.539084] do_syscall_64+0x62/0x170
[ 1721.539088] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1721.539090] RIP: 0033:0x7f2fc4d1fdf5
[ 1721.539092] RSP: 002b:00007ffc726f6dc8 EFLAGS: 00000246 ORIG_RAX:
0000000000000000
[ 1721.539094] RAX: ffffffffffffffda RBX: 00007f2fc6213678 RCX:
00007f2fc4d1fdf5
[ 1721.539095] RDX: 0000000000000400 RSI: 000055955b781a10 RDI:
0000000000000007
[ 1721.539096] RBP: 00007f2fc6213670 R08: 0000000000000000 R09:
00007f2fc4d80060
[ 1721.539098] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000007
[ 1721.539099] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 1721.539103] </TASK>
[ 1721.539104] Future hung task reports are suppressed, see sysctl
kernel.hung_task_warnings
[ 1776.268834] watchdog: BUG: soft lockup - CPU#10 stuck for 23s!
[osms-agent:3184]
[ 1776.268839] Modules linked in: binfmt_misc ip6t_REJECT ipt_REJECT
xt_comment xt_owner nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6
nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
rfkill nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
ip_set cuse fuse vfat fat intel_rapl_msr intel_rapl_common raid0 kvm_amd
mlx5_ib bochs ccp drm_vram_helper drm_ttm_helper ib_uverbs mlx5_vdpa ttm
vringh drm_kms_helper kvm vhost_iotlb ib_core vdpa drm joydev pcspkr
i2c_piix4 pvpanic_mmio pvpanic i2c_smbus xfs iscsi_tcp libiscsi_tcp
libiscsi nvme_tcp nvme_fabrics nvme nvme_core sd_mod sg virtio_scsi
mlx5_core ata_generic pata_acpi crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel sha512_ssse3 ata_piix mlxfw sha256_ssse3 sha1_ssse3
psample libata virtio_pci tls serio_raw virtio_pci_legacy_dev
virtio_pci_modern_dev pci_hyperv_intf qemu_fw_cfg dm_multipath sunrpc
dm_mirror dm_region_hash dm_log dm_mod scsi_transport_iscsi aesni_intel
crypto_simd cryptd
[ 1776.268933] CPU: 10 UID: 0 PID: 3184 Comm: osms-agent Tainted: G
W 6.12.0-rc4-g556c97f2ecbf #40
[ 1776.268937] Tainted: [W]=WARN
[ 1776.268939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1776.268940] RIP: 0010:smp_call_function_many_cond+0x312/0x4e0
[ 1776.268947] Code: 48 00 39 05 f0 74 fe 01 48 89 c5 0f 86 79 ff ff ff
48 63 c5 48 8b 3b 48 03 3c c5 00 8e 5a 95 66 90 8b 47 08 a8 01 74 09 f3
90 <8b> 47 08 a8 01 75 f7 83 c5 01 eb ba e8 cd f6 ff ff eb f4 83 7c 24
[ 1776.268949] RSP: 0018:ffffc0c8c2d9fc38 EFLAGS: 00000202
[ 1776.268951] RAX: 0000000000000011 RBX: ffff9ff73fb37900 RCX:
0000000000000000
[ 1776.268953] RDX: 0000000000000001 RSI: 0000000000000010 RDI:
ffff9ff73f6bd340
[ 1776.268954] RBP: 0000000000000001 R08: 0000000000000001 R09:
ffff9ff73f6bd340
[ 1776.268956] R10: 0000000000000002 R11: 0000000000000000 R12:
0000000000000001
[ 1776.268957] R13: 0000000000000001 R14: 0000000000000010 R15:
ffff9ff73f6bd340
[ 1776.268969] FS: 00007fdbcd600700(0000) GS:ffff9ff73fb00000(0000)
knlGS:0000000000000000
[ 1776.268971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1776.268973] CR2: 00007fdbea983560 CR3: 00000001095f2005 CR4:
0000000000770ef0
[ 1776.268976] PKRU: 55555554
[ 1776.268977] Call Trace:
[ 1776.268980] <IRQ>
[ 1776.268982] ? watchdog_timer_fn+0x1e2/0x260
[ 1776.268987] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 1776.268989] ? __hrtimer_run_queues+0x10c/0x270
[ 1776.268995] ? hrtimer_interrupt+0x109/0x250
[ 1776.268997] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1776.269002] ? __sysvec_apic_timer_interrupt+0x55/0x120
[ 1776.269006] ? sysvec_apic_timer_interrupt+0x6c/0x90
[ 1776.269010] </IRQ>
[ 1776.269012] <TASK>
[ 1776.269013] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 1776.269019] ? smp_call_function_many_cond+0x312/0x4e0
[ 1776.269021] ? smp_call_function_many_cond+0x2ea/0x4e0
[ 1776.269023] ? __pfx_flush_tlb_func+0x10/0x10
[ 1776.269029] on_each_cpu_cond_mask+0x29/0x50
[ 1776.269031] flush_tlb_mm_range+0x13f/0x160
[ 1776.269035] dup_mmap+0x2ea/0x740
[ 1776.269042] copy_process+0x12b2/0x1d20
[ 1776.269047] kernel_clone+0x9e/0x390
[ 1776.269050] __do_sys_clone+0x66/0x90
[ 1776.269055] do_syscall_64+0x62/0x170
[ 1776.269059] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1776.269063] RIP: 0033:0x7fdbeb4f9a73
[ 1776.269066] Code: db 0f 85 28 01 00 00 64 4c 8b 0c 25 10 00 00 00 45
31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f
05 <48> 3d 00 f0 ff ff 0f 87 b9 00 00 00 41 89 c5 85 c0 0f 85 c6 00 00
[ 1776.269067] RSP: 002b:00007fdbcd5fe660 EFLAGS: 00000246 ORIG_RAX:
0000000000000038
[ 1776.269070] RAX: ffffffffffffffda RBX: 0000000000000001 RCX:
00007fdbeb4f9a73
[ 1776.269071] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000001200011
[ 1776.269072] RBP: 0000000000000001 R08: 0000000000000000 R09:
00007fdbcd600700
[ 1776.269074] R10: 00007fdbcd6009d0 R11: 0000000000000246 R12:
0000000000000004
[ 1776.269075] R13: 00007fdbeb309b70 R14: 00007fdbcd5fe768 R15:
00007fdbd7cad2e8
[ 1776.269079] </TASK>
[ 1776.269081] Kernel panic - not syncing: softlockup: hung tasks
[ 1776.269083] CPU: 10 UID: 0 PID: 3184 Comm: osms-agent Tainted: G
W L 6.12.0-rc4-g556c97f2ecbf #40
[ 1776.269086] Tainted: [W]=WARN, [L]=SOFTLOCKUP
[ 1776.269087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.5.1 06/16/2021
[ 1776.269088] Call Trace:
[ 1776.269090] <IRQ>
[ 1776.269091] panic+0x34f/0x380
[ 1776.269095] watchdog_timer_fn+0x220/0x260
[ 1776.269098] ? __pfx_watchdog_timer_fn+0x10/0x10
[ 1776.269100] __hrtimer_run_queues+0x10c/0x270
[ 1776.269104] hrtimer_interrupt+0x109/0x250
[ 1776.269106] ? srso_alias_return_thunk+0x5/0xfbef5
[ 1776.269109] __sysvec_apic_timer_interrupt+0x55/0x120
[ 1776.269112] sysvec_apic_timer_interrupt+0x6c/0x90
[ 1776.269115] </IRQ>
[ 1776.269116] <TASK>
[ 1776.269118] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 1776.269120] RIP: 0010:smp_call_function_many_cond+0x312/0x4e0
[ 1776.269122] Code: 48 00 39 05 f0 74 fe 01 48 89 c5 0f 86 79 ff ff ff
48 63 c5 48 8b 3b 48 03 3c c5 00 8e 5a 95 66 90 8b 47 08 a8 01 74 09 f3
90 <8b> 47 08 a8 01 75 f7 83 c5 01 eb ba e8 cd f6 ff ff eb f4 83 7c 24
[ 1776.269124] RSP: 0018:ffffc0c8c2d9fc38 EFLAGS: 00000202
[ 1776.269126] RAX: 0000000000000011 RBX: ffff9ff73fb37900 RCX:
0000000000000000
[ 1776.269127] RDX: 0000000000000001 RSI: 0000000000000010 RDI:
ffff9ff73f6bd340
[ 1776.269128] RBP: 0000000000000001 R08: 0000000000000001 R09:
ffff9ff73f6bd340
[ 1776.269129] R10: 0000000000000002 R11: 0000000000000000 R12:
0000000000000001
[ 1776.269130] R13: 0000000000000001 R14: 0000000000000010 R15:
ffff9ff73f6bd340
[ 1776.269135] ? smp_call_function_many_cond+0x2ea/0x4e0
[ 1776.269137] ? __pfx_flush_tlb_func+0x10/0x10
[ 1776.269141] on_each_cpu_cond_mask+0x29/0x50
[ 1776.269143] flush_tlb_mm_range+0x13f/0x160
[ 1776.269146] dup_mmap+0x2ea/0x740
[ 1776.269152] copy_process+0x12b2/0x1d20
[ 1776.269156] kernel_clone+0x9e/0x390
[ 1776.269160] __do_sys_clone+0x66/0x90
[ 1776.269164] do_syscall_64+0x62/0x170
[ 1776.269167] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 1776.269169] RIP: 0033:0x7fdbeb4f9a73
[ 1776.269171] Code: db 0f 85 28 01 00 00 64 4c 8b 0c 25 10 00 00 00 45
31 c0 4d 8d 91 d0 02 00 00 31 d2 31 f6 bf 11 00 20 01 b8 38 00 00 00 0f
05 <48> 3d 00 f0 ff ff 0f 87 b9 00 00 00 41 89 c5 85 c0 0f 85 c6 00 00
[ 1776.269172] RSP: 002b:00007fdbcd5fe660 EFLAGS: 00000246 ORIG_RAX:
0000000000000038
[ 1776.269174] RAX: ffffffffffffffda RBX: 0000000000000001 RCX:
00007fdbeb4f9a73
[ 1776.269175] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000001200011
[ 1776.269177] RBP: 0000000000000001 R08: 0000000000000000 R09:
00007fdbcd600700
[ 1776.269178] R10: 00007fdbcd6009d0 R11: 0000000000000246 R12:
0000000000000004
[ 1776.269179] R13: 00007fdbeb309b70 R14: 00007fdbcd5fe768 R15:
00007fdbd7cad2e8
[ 1776.269183] </TASK>
[ 1776.272383] Kernel Offset: 0x12c00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1776.712795] ---[ end Kernel panic - not syncing: softlockup: hung
tasks ]---
