Hi,
I'm getting soft lockups when running the BPF arena selftests on powerpc
(ppcle64). The issue is 100% reproducible on the latest bpf-next with
`./test_progs -t arena`.
A console snippet for one CPU lockup looks like this:
[ 1124.671746] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u34:0:58]
[ 1124.675554] CPU#1 Utilization every 4s during lockup:
[ 1124.675584] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675621] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675659] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675733] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675770] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.675921] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OE 6.12.0-rc4+ #1
[ 1124.675975] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 1124.676005] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.676063] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.676101] NIP: c000000000551d3c LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.676145] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OE (6.12.0-rc4+)
[ 1124.676189] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44082828 XER: 00000000
[ 1124.676251] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.676251] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1124.676251] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1124.676251] GPR08: 0000000000000000 0000000000000000 0000000058006001 0000000024082828
[ 1124.676251] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1124.676251] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1124.676251] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1124.676251] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.676251] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00bf073759e8000
[ 1124.676627] NIP [c000000000551d3c] __apply_to_page_range+0x55c/0xea0
[ 1124.676667] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.676706] Call Trace:
[ 1124.676730] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.676784] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.676824] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.676870] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.676915] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.676954] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.676992] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.677038] Code: 60420000 7d29a039 4082ff00 3fff0001 7c3cf840 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e <5106421e> 50e9421e 5106463e 714a0040
There are more CPUs affected, the full console log is attached.
Thanks.
Viktor
[ 1099.746415] bpf_testmod: loading out-of-tree module taints kernel.
[ 1099.750657] bpf_testmod: module verification failed: signature and/or required key missing - tainting kernel
[ 1124.671746] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u34:0:58]
[ 1124.675554] CPU#1 Utilization every 4s during lockup:
[ 1124.675584] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675621] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675659] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675696] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675733] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.675770] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.675921] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OE 6.12.0-rc4+ #1
[ 1124.675975] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 1124.676005] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.676063] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.676101] NIP: c000000000551d3c LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.676145] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OE (6.12.0-rc4+)
[ 1124.676189] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44082828 XER: 00000000
[ 1124.676251] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.676251] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1124.676251] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1124.676251] GPR08: 0000000000000000 0000000000000000 0000000058006001 0000000024082828
[ 1124.676251] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1124.676251] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1124.676251] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1124.676251] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.676251] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00bf073759e8000
[ 1124.676627] NIP [c000000000551d3c] __apply_to_page_range+0x55c/0xea0
[ 1124.676667] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.676706] Call Trace:
[ 1124.676730] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.676784] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.676824] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.676870] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.676915] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.676954] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.676992] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.677038] Code: 60420000 7d29a039 4082ff00 3fff0001 7c3cf840 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e <5106421e> 50e9421e 5106463e 714a0040
[ 1124.801744] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u37:7:34302]
[ 1124.805191] CPU#4 Utilization every 4s during lockup:
[ 1124.805221] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805260] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805297] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805335] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805373] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.805410] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.805553] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1124.805607] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1124.805643] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.805701] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.805738] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.805782] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1124.805826] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1124.805887] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.805887] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1124.805887] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1124.805887] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1124.805887] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1124.805887] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1124.805887] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1124.805887] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.805887] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c00bf743dfb38000
[ 1124.806260] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1124.806301] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.806339] Call Trace:
[ 1124.806354] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.806408] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.806448] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.806494] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.806540] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.806579] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.806617] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.806662] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1124.821743] watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u34:3:47428]
[ 1124.821809] CPU#5 Utilization every 4s during lockup:
[ 1124.821842] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.821883] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.821923] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.821962] #4: 101% system, 0% softirq, 1% hardirq, 0% idle
[ 1124.822001] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1124.822041] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1124.822189] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1124.822245] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1124.822284] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1124.822343] Workqueue: events_unbound bpf_map_free_deferred
[ 1124.822381] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1124.822428] REGS: c0000000234dfa20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1124.822474] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1124.822538] CFAR: 0000000000000000 IRQMASK: 0
[ 1124.822538] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1124.822538] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1124.822538] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1124.822538] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1124.822538] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1124.822538] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1124.822538] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1124.822538] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c00bf7e896038000
[ 1124.822930] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1124.822973] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1124.823014] Call Trace:
[ 1124.823030] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1124.823087] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1124.823130] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1124.823179] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1124.823229] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1124.823271] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1124.823311] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1124.823360] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1148.671553] watchdog: BUG: soft lockup - CPU#1 stuck for 46s! [kworker/u34:0:58]
[ 1148.675300] CPU#1 Utilization every 4s during lockup:
[ 1148.675330] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675375] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675414] #3: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675454] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675493] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.675533] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1148.675696] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OEL 6.12.0-rc4+ #1
[ 1148.675758] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1148.675797] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1148.675863] Workqueue: events_unbound bpf_map_free_deferred
[ 1148.675910] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1148.675958] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1148.676010] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082828 XER: 00000000
[ 1148.676080] CFAR: 0000000000000000 IRQMASK: 0
[ 1148.676080] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1148.676080] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1148.676080] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082828
[ 1148.676080] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1148.676080] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1148.676080] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1148.676080] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1148.676080] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c00fc45376338000
[ 1148.676496] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1148.676544] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1148.676593] Call Trace:
[ 1148.676611] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1148.676677] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1148.676720] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1148.676773] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1148.676824] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1148.676874] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1148.676920] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1148.676970] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1148.801551] watchdog: BUG: soft lockup - CPU#4 stuck for 46s! [kworker/u37:7:34302]
[ 1148.805287] CPU#4 Utilization every 4s during lockup:
[ 1148.805345] #1: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805387] #2: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805429] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805467] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805507] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.805546] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1148.805696] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1148.805754] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1148.805792] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1148.805853] Workqueue: events_unbound bpf_map_free_deferred
[ 1148.805890] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1148.805937] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1148.805983] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1148.806046] CFAR: 0000000000000000 IRQMASK: 0
[ 1148.806046] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1148.806046] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1148.806046] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1148.806046] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1148.806046] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1148.806046] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1148.806046] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1148.806046] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c00fcadcfbb08000
[ 1148.806439] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1148.806481] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1148.806523] Call Trace:
[ 1148.806539] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1148.806597] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1148.806639] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1148.806688] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1148.806738] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1148.806780] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1148.806821] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1148.806869] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1148.821550] watchdog: BUG: soft lockup - CPU#5 stuck for 46s! [kworker/u34:3:47428]
[ 1148.821614] CPU#5 Utilization every 4s during lockup:
[ 1148.821651] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821693] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821752] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821794] #4: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821846] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1148.821886] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1148.822059] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1148.822134] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1148.822187] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1148.822260] Workqueue: events_unbound bpf_map_free_deferred
[ 1148.822296] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1148.822357] REGS: c0000000234dfa20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1148.822402] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1148.822483] CFAR: 0000000000000000 IRQMASK: 0
[ 1148.822483] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1148.822483] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1148.822483] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1148.822483] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1148.822483] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1148.822483] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1148.822483] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1148.822483] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c00fcbe7f3158000
[ 1148.822964] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1148.823023] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1148.823065] Call Trace:
[ 1148.823081] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1148.823157] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1148.823214] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1148.823263] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1148.823329] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1148.823388] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1148.823429] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1148.823491] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1159.951465] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1159.954380] rcu: 4-....: (5999 ticks this GP) idle=788c/1/0x4000000000000002 softirq=64429/64430 fqs=2997
[ 1159.954436] rcu: (t=6000 jiffies g=36285 q=1497 ncpus=8)
[ 1159.954471] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1159.954476] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1159.954478] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1159.954481] Workqueue: events_unbound bpf_map_free_deferred
[ 1159.954491] NIP: c000000000551d60 LR: c000000000551c30 CTR: c0000000004733b0
[ 1159.954494] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1159.954496] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1159.954506] CFAR: 0000000000000000 IRQMASK: 0
[ 1159.954506] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1159.954506] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1159.954506] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1159.954506] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1159.954506] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1159.954506] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1159.954506] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1159.954506] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c01191c4d0018000
[ 1159.954536] NIP [c000000000551d60] __apply_to_page_range+0x580/0xea0
[ 1159.954541] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1159.954545] Call Trace:
[ 1159.954546] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1159.954551] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1159.954555] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1159.954559] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1159.954564] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1159.954567] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1159.954570] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1159.954573] Code: 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e 78c9000e 4082ffbc 7d29c839 4082feb8 <3fff0001> 7c3fe040 4082ffbc 4bfffedc
[ 1159.954586] Sending NMI from CPU 4 to CPUs 5:
[ 1159.954604] NMI backtrace for cpu 5
[ 1159.956027] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1159.956102] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1159.956141] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1159.956202] Workqueue: events_unbound bpf_map_free_deferred
[ 1159.956253] NIP: c000000000551d60 LR: c000000000551c30 CTR: c0000000004733b0
[ 1159.956300] REGS: c0000000234dfa20 TRAP: 0500 Tainted: G OEL (6.12.0-rc4+)
[ 1159.956346] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1159.956410] CFAR: 0000000000000000 IRQMASK: 0
[ 1159.956410] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1159.956410] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1159.956410] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1159.956410] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1159.956410] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1159.956410] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1159.956410] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1159.956410] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c0119273ac8f8000
[ 1159.956800] NIP [c000000000551d60] __apply_to_page_range+0x580/0xea0
[ 1159.956858] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1159.956898] Call Trace:
[ 1159.956915] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1159.956987] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1159.957029] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1159.957087] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1159.957135] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1159.957176] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1159.957217] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1159.957265] Code: 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e 78c9000e 4082ffbc 7d29c839 4082feb8 <3fff0001> 7c3fe040 4082ffbc 4bfffedc
[ 1172.671360] watchdog: BUG: soft lockup - CPU#1 stuck for 68s! [kworker/u34:0:58]
[ 1172.673229] CPU#1 Utilization every 4s during lockup:
[ 1172.673259] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673298] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673337] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673377] #4: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.673417] #5: 100% system, 0% softirq, 1% hardirq, 0% idle
[ 1172.673456] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1172.673611] CPU: 1 UID: 0 PID: 58 Comm: kworker/u34:0 Tainted: G OEL 6.12.0-rc4+ #1
[ 1172.673673] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1172.673711] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1172.673772] Workqueue: events_unbound bpf_map_free_deferred
[ 1172.673816] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1172.673863] REGS: c000000008a37a20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1172.673909] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082828 XER: 00000000
[ 1172.673974] CFAR: 0000000000000000 IRQMASK: 0
[ 1172.673974] GPR00: c000000000551c30 c000000008a37cc0 c00000000214f800 0000000000000000
[ 1172.673974] GPR04: 000000000000003b c00c00000044e3c8 0000000000000000 0000000000000000
[ 1172.673974] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082828
[ 1172.673974] GPR12: c0000000004733b0 c00000003ffff480 c0000000043cb7c0 c0000000043b1028
[ 1172.673974] GPR16: c008000305f78000 0000000000000000 0000000000000001 0000000000000000
[ 1172.673974] GPR20: fffffffffffffe7f c008000305f77fff c000000003cbe780 c000000001b26120
[ 1172.673974] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1172.673974] GPR28: c008000206000000 0000000000000000 c0000000004733b0 c013980b18d48000
[ 1172.674365] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1172.674409] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1172.674450] Call Trace:
[ 1172.674467] [c000000008a37cc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1172.674524] [c000000008a37de0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1172.674567] [c000000008a37e10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1172.674616] [c000000008a37e40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1172.674665] [c000000008a37ef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1172.674707] [c000000008a37f90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1172.674746] [c000000008a37fe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1172.674795] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
[ 1172.821358] watchdog: BUG: soft lockup - CPU#5 stuck for 68s! [kworker/u34:3:47428]
[ 1172.821774] CPU#5 Utilization every 4s during lockup:
[ 1172.821803] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821841] #2: 100% system, 0% softirq, 1% hardirq, 0% idle
[ 1172.821879] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821919] #4: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821959] #5: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1172.821998] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1172.822141] CPU: 5 UID: 0 PID: 47428 Comm: kworker/u34:3 Tainted: G OEL 6.12.0-rc4+ #1
[ 1172.822197] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1172.822235] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1172.822295] Workqueue: events_unbound bpf_map_free_deferred
[ 1172.822331] NIP: c000000000551d60 LR: c000000000551c30 CTR: c0000000004733b0
[ 1172.822378] REGS: c0000000234dfa20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1172.822424] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082808 XER: 00000000
[ 1172.822487] CFAR: 0000000000000000 IRQMASK: 0
[ 1172.822487] GPR00: c000000000551c30 c0000000234dfcc0 c00000000214f800 0000000000000000
[ 1172.822487] GPR04: 0000000000000026 c00c000000163708 0000000000000000 0000000000000000
[ 1172.822487] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082808
[ 1172.822487] GPR12: c0000000004733b0 c00000003ffdb080 c0000000043ca7b0 c0000000043b0828
[ 1172.822487] GPR16: c008000205f58000 0000000000000000 0000000000000001 0000000000000000
[ 1172.822487] GPR20: fffffffffffffe7f c008000205f57fff c000000003cbe780 c000000001b26120
[ 1172.822487] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1172.822487] GPR28: c008000106000000 0000000000000000 c0000000004733b0 c0139fb865828000
[ 1172.822883] NIP [c000000000551d60] __apply_to_page_range+0x580/0xea0
[ 1172.822925] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1172.822966] Call Trace:
[ 1172.822982] [c0000000234dfcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1172.823039] [c0000000234dfde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1172.823081] [c0000000234dfe10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1172.823129] [c0000000234dfe40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1172.823178] [c0000000234dfef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1172.823219] [c0000000234dff90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1172.823260] [c0000000234dffe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1172.823308] Code: 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e 78c9000e 4082ffbc 7d29c839 4082feb8 <3fff0001> 7c3fe040 4082ffbc 4bfffedc
[ 1184.801263] watchdog: BUG: soft lockup - CPU#4 stuck for 79s! [kworker/u37:7:34302]
[ 1184.804034] CPU#4 Utilization every 4s during lockup:
[ 1184.804111] #1: 101% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804150] #2: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804188] #3: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804226] #4: 101% system, 0% softirq, 1% hardirq, 0% idle
[ 1184.804263] #5: 100% system, 0% softirq, 0% hardirq, 0% idle
[ 1184.804312] Modules linked in: bpf_testmod(OE) bonding tls rfkill virtio_net net_failover vmx_crypto failover virtio_balloon crct10dif_vpmsum fuse loop nfnetlink zram vsock_loopback vmw_vsock_virtio_transport_common vsock virtio_blk crc32c_vpmsum virtio_console
[ 1184.804525] CPU: 4 UID: 0 PID: 34302 Comm: kworker/u37:7 Tainted: G OEL 6.12.0-rc4+ #1
[ 1184.804584] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [L]=SOFTLOCKUP
[ 1184.804626] Hardware name: IBM pSeries (emulated by qemu) POWER8E (raw) 0x4b0201 of:SLOF,HEAD hv:linux,kvm pSeries
[ 1184.804731] Workqueue: events_unbound bpf_map_free_deferred
[ 1184.804811] NIP: c000000000551d50 LR: c000000000551c30 CTR: c0000000004733b0
[ 1184.804856] REGS: c00000011401ba20 TRAP: 0900 Tainted: G OEL (6.12.0-rc4+)
[ 1184.804899] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24082282 XER: 00000000
[ 1184.804988] CFAR: 0000000000000000 IRQMASK: 0
[ 1184.804988] GPR00: c000000000551c30 c00000011401bcc0 c00000000214f800 0000000000000000
[ 1184.804988] GPR04: 0000000000000124 c00c00000044e408 0000000000000000 0000000000000000
[ 1184.804988] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000024082282
[ 1184.804988] GPR12: c0000000004733b0 c00000003fffc480 c0000000043c87a0 c0000000043b0028
[ 1184.804988] GPR16: c008000105f38000 0000000000000000 0000000000000001 0000000000000000
[ 1184.804988] GPR20: fffffffffffffe7f c008000105f37fff c000000003cbe780 c000000001b26120
[ 1184.804988] GPR24: c000000003da0380 ff7fffffffffefbf c000000003cbe780 0000000000000001
[ 1184.804988] GPR28: c008000006000000 0000000000000000 c0000000004733b0 c015882289c58000
[ 1184.805363] NIP [c000000000551d50] __apply_to_page_range+0x570/0xea0
[ 1184.805410] LR [c000000000551c30] __apply_to_page_range+0x450/0xea0
[ 1184.805448] Call Trace:
[ 1184.805464] [c00000011401bcc0] [c000000000551c30] __apply_to_page_range+0x450/0xea0 (unreliable)
[ 1184.805525] [c00000011401bde0] [c000000000473360] arena_map_free+0x70/0xc0
[ 1184.805564] [c00000011401be10] [c0000000003ee324] bpf_map_free_deferred+0x94/0x110
[ 1184.805610] [c00000011401be40] [c00000000019bf8c] process_one_work+0x1fc/0x520
[ 1184.805662] [c00000011401bef0] [c00000000019d96c] worker_thread+0x33c/0x4f0
[ 1184.805707] [c00000011401bf90] [c0000000001aa7a4] kthread+0x134/0x140
[ 1184.805745] [c00000011401bfe0] [c00000000000dd58] start_kernel_thread+0x14/0x18
[ 1184.805791] Code: 4182ff28 e9370000 e90e0000 81490020 79070022 5506c03e 54e9c03e 5106421e 50e9421e 5106463e 714a0040 50e9463e <78c9000e> 4082ffbc 7d29c839 4082feb8
