On Mon, Jul 08, 2024 at 11:16:48PM -0400, matoro wrote: > On 2024-07-05 16:34, Vitaly Chikunov wrote: > > Hi, > > > > There is new WARNING and Oops on ppc64le in v6.6.37 when running LTP tests: > > bpf_prog01, bpf_prog02, bpf_prog04, bpf_prog05, prctl04. Logs excerpt > > below. I > > see there is 1 commit in v6.6.36..v6.6.37 with call to > > bpf_jit_binary_pack_finalize, backported from 5 patch mainline patchset: > > > > f99feda5684a powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free] > > > > Log: > > > > [ 8.822079] LTP: starting bpf_prog01 > > [ 8.841853] ------------[ cut here ]------------ > > [ 8.841946] Trying to vfree() bad address (00000000453be747) > > [ 8.842024] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2700 > > remove_vm_area+0xb4/0xf0 > > [ 8.842103] Modules linked in: virtio_rng rng_core virtio_net > > net_failover failover sd_mod ata_generic ata_piix libata scsi_mod > > scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev > > virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs > > [ 8.842323] CPU: 6 PID: 689 Comm: bpf_prog01 Not tainted > > 6.6.37-un-def-alt1 #1 > > [ 8.842396] Hardware name: IBM pSeries (emulated by qemu) POWER8 > > (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries > > [ 8.842519] NIP: c0000000004faf04 LR: c0000000004faf00 CTR: > > 0000000000000000 > > [ 8.842598] REGS: c000000009b6f250 TRAP: 0700 Not tainted > > (6.6.37-un-def-alt1) > > [ 8.842669] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: > > 28002822 XER: 00000000 > > [ 8.842748] CFAR: c00000000015df94 IRQMASK: 0 > > [ 8.842748] GPR00: 0000000000000000 c000000009b6f4f0 > > c000000001ac7f00 0000000000000000 > > [ 8.842748] GPR04: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.842748] GPR08: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.842748] GPR12: 0000000000000000 c00000003fff7a00 > > 0000000000000000 0000000000000000 > > [ 8.842748] GPR16: 0000000000000012 0000000000000000 > > 000000000000008c 0000000000000000 > > [ 8.842748] GPR20: c008000000040a40 0000000000000002 > > c0000000022a7560 c008000000040a4c > > [ 8.842748] GPR24: c000000005716480 0000000000000000 > > c000000002155698 c0000000022a7680 > > [ 8.842748] GPR28: c000000002155688 c008000000040a40 > > c008000000040a40 c008000000040a40 > > [ 8.843347] NIP [c0000000004faf04] remove_vm_area+0xb4/0xf0 > > [ 8.843398] LR [c0000000004faf00] remove_vm_area+0xb0/0xf0 > > [ 8.843448] Call Trace: > > [ 8.843484] [c000000009b6f4f0] [c0000000004faf00] > > remove_vm_area+0xb0/0xf0 (unreliable) > > [ 8.843559] [c000000009b6f560] [c0000000004fb360] vfree+0x60/0x2a0 > > [ 8.843621] [c000000009b6f5e0] [c000000000269c6c] > > module_memfree+0x3c/0x60 > > [ 8.843685] [c000000009b6f600] [c00000000038cf60] > > bpf_jit_free_exec+0x20/0x40 > > [ 8.843759] [c000000009b6f620] [c00000000038f518] > > bpf_prog_pack_free+0x2f8/0x390 > > [ 8.843832] [c000000009b6f6b0] [c00000000038f878] > > bpf_jit_binary_pack_finalize+0x98/0xd0 > > [ 8.843906] [c000000009b6f6e0] [c000000000118240] > > bpf_int_jit_compile+0x2c0/0x710 > > [ 8.843979] [c000000009b6f830] [c00000000038ef64] > > bpf_prog_select_runtime+0x154/0x1b0 > > [ 8.844053] [c000000009b6f880] [c000000000398edc] > > bpf_prog_load+0x94c/0xe90 > > [ 8.844114] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 > > [ 8.844176] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 > > [ 8.844237] [c000000009b6fae0] [c000000000030230] > > system_call_exception+0x190/0x390 > > [ 8.844312] [c000000009b6fe50] [c00000000000c7d4] > > system_call_common+0xf4/0x258 > > [ 8.844386] --- interrupt: c00 at 0x7fffb0839ad4 > > [ 8.844437] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: > > 0000000000000000 > > [ 8.844524] REGS: c000000009b6fe80 TRAP: 0c00 Not tainted > > (6.6.37-un-def-alt1) > > [ 8.844596] MSR: 800000000280f033 > > <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 > > [ 8.844690] IRQMASK: 0 > > [ 8.844690] GPR00: 0000000000000169 00007fffd8534200 > > 00007fffb0936d00 0000000000000005 > > [ 8.844690] GPR04: 00007fffb06aff90 0000000000000070 > > 000000012a0538a0 0000000000000001 > > [ 8.844690] GPR08: 000000012a0801f4 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.844690] GPR12: 0000000000000000 00007fffb09ea540 > > 0000000000000000 0000000000000000 > > [ 8.844690] GPR16: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.844690] GPR20: 00007fffd85344b0 0000000000000000 > > 0000000000000001 0000000000000000 > > [ 8.844690] GPR24: 000000012a0801f4 00007fffb06ce000 > > 0000000000000000 00000000000f4240 > > [ 8.844690] GPR28: 00007fffb06aff90 00007fffb09e3550 > > 0000000000000001 0000000000001118 > > [ 8.845267] NIP [00007fffb0839ad4] 0x7fffb0839ad4 > > [ 8.845315] LR [000000012a027fb4] 0x12a027fb4 > > [ 8.845363] --- interrupt: c00 > > [ 8.845399] Code: 38000000 38800000 39200000 4e800020 60000000 > > 60000000 60420000 3c62ffa2 7fe4fb78 3863e698 4bc62f8d 60000000 > > <0fe00000> 38210070 3bc00000 e8010010 > > [ 8.845550] ---[ end trace 0000000000000000 ]--- > > [ 8.845603] ------------[ cut here ]------------ > > [ 8.845651] Trying to vfree() nonexistent vm area (00000000453be747) > > [ 8.845714] WARNING: CPU: 6 PID: 689 at mm/vmalloc.c:2835 > > vfree+0x1d8/0x2a0 > > [ 8.845776] Modules linked in: virtio_rng rng_core virtio_net > > net_failover failover sd_mod ata_generic ata_piix libata scsi_mod > > scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev > > virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs > > [ 8.845989] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W > > 6.6.37-un-def-alt1 #1 > > [ 8.846072] Hardware name: IBM pSeries (emulated by qemu) POWER8 > > (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries > > [ 8.846177] NIP: c0000000004fb4d8 LR: c0000000004fb4d4 CTR: > > 0000000000000000 > > [ 8.846248] REGS: c000000009b6f2c0 TRAP: 0700 Tainted: G W > > (6.6.37-un-def-alt1) > > [ 8.846330] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: > > 28002222 XER: 00000000 > > [ 8.846408] CFAR: c00000000015df94 IRQMASK: 0 > > [ 8.846408] GPR00: 0000000000000000 c000000009b6f560 > > c000000001ac7f00 0000000000000000 > > [ 8.846408] GPR04: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.846408] GPR08: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.846408] GPR12: 0000000000000000 c00000003fff7a00 > > 0000000000000000 0000000000000000 > > [ 8.846408] GPR16: 0000000000000012 0000000000000000 > > 000000000000008c 0000000000000000 > > [ 8.846408] GPR20: c008000000040a40 0000000000000002 > > c0000000022a7560 c008000000040a4c > > [ 8.846408] GPR24: c000000005716480 0000000000000000 > > c000000002155698 c0000000022a7680 > > [ 8.846408] GPR28: c000000002155688 0000000000000000 > > c008000000040a40 0000000000000000 > > [ 8.851030] NIP [c0000000004fb4d8] vfree+0x1d8/0x2a0 > > [ 8.851085] LR [c0000000004fb4d4] vfree+0x1d4/0x2a0 > > [ 8.851135] Call Trace: > > [ 8.851160] [c000000009b6f560] [c0000000004fb4d4] vfree+0x1d4/0x2a0 > > (unreliable) > > [ 8.851234] [c000000009b6f5e0] [c000000000269c6c] > > module_memfree+0x3c/0x60 > > [ 8.851297] [c000000009b6f600] [c00000000038cf60] > > bpf_jit_free_exec+0x20/0x40 > > [ 8.851371] [c000000009b6f620] [c00000000038f518] > > bpf_prog_pack_free+0x2f8/0x390 > > [ 8.851445] [c000000009b6f6b0] [c00000000038f878] > > bpf_jit_binary_pack_finalize+0x98/0xd0 > > [ 8.851529] [c000000009b6f6e0] [c000000000118240] > > bpf_int_jit_compile+0x2c0/0x710 > > [ 8.851602] [c000000009b6f830] [c00000000038ef64] > > bpf_prog_select_runtime+0x154/0x1b0 > > [ 8.851675] [c000000009b6f880] [c000000000398edc] > > bpf_prog_load+0x94c/0xe90 > > [ 8.851737] [c000000009b6f990] [c00000000039c878] __sys_bpf+0x418/0x2970 > > [ 8.851798] [c000000009b6fac0] [c00000000039f1a0] sys_bpf+0x30/0x50 > > [ 8.851860] [c000000009b6fae0] [c000000000030230] > > system_call_exception+0x190/0x390 > > [ 8.851934] [c000000009b6fe50] [c00000000000c7d4] > > system_call_common+0xf4/0x258 > > [ 8.852007] --- interrupt: c00 at 0x7fffb0839ad4 > > [ 8.852057] NIP: 00007fffb0839ad4 LR: 000000012a027fb4 CTR: > > 0000000000000000 > > [ 8.852128] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W > > (6.6.37-un-def-alt1) > > [ 8.852212] MSR: 800000000280f033 > > <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 22002240 XER: 00000000 > > [ 8.852307] IRQMASK: 0 > > [ 8.852307] GPR00: 0000000000000169 00007fffd8534200 > > 00007fffb0936d00 0000000000000005 > > [ 8.852307] GPR04: 00007fffb06aff90 0000000000000070 > > 000000012a0538a0 0000000000000001 > > [ 8.852307] GPR08: 000000012a0801f4 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.852307] GPR12: 0000000000000000 00007fffb09ea540 > > 0000000000000000 0000000000000000 > > [ 8.852307] GPR16: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.852307] GPR20: 00007fffd85344b0 0000000000000000 > > 0000000000000001 0000000000000000 > > [ 8.852307] GPR24: 000000012a0801f4 00007fffb06ce000 > > 0000000000000000 00000000000f4240 > > [ 8.852307] GPR28: 00007fffb06aff90 00007fffb09e3550 > > 0000000000000001 0000000000001118 > > [ 8.852889] NIP [00007fffb0839ad4] 0x7fffb0839ad4 > > [ 8.852938] LR [000000012a027fb4] 0x12a027fb4 > > [ 8.852986] --- interrupt: c00 > > [ 8.853022] Code: 4e800020 60420000 3949ffff 4bffff0c 38210080 > > ebe1fff8 4bfffd68 3c62ffa2 7fc4f378 3863e6f0 4bc629b9 60000000 > > <0fe00000> eba10068 4bffff8c 2c080000 > > [ 8.853164] ---[ end trace 0000000000000000 ]--- > > [ 8.856619] kernel tried to execute exec-protected page > > (c008000000040a4c) - exploit attempt? (uid: 0) > > [ 8.856717] BUG: Unable to handle kernel instruction fetch > > [ 8.856763] Faulting instruction address: 0xc008000000040a4c > > [ 8.856825] Oops: Kernel access of bad area, sig: 11 [#1] > > [ 8.856875] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries > > [ 8.856937] Modules linked in: virtio_rng rng_core virtio_net > > net_failover failover sd_mod ata_generic ata_piix libata scsi_mod > > scsi_common virtio_blk virtio_pci virtio_pci_legacy_dev > > virtio_pci_modern_dev 9pnet_virtio virtio_ring virtio 9p 9pnet netfs > > [ 8.857154] CPU: 6 PID: 689 Comm: bpf_prog01 Tainted: G W > > 6.6.37-un-def-alt1 #1 > > [ 8.857236] Hardware name: IBM pSeries (emulated by qemu) POWER8 > > (raw) 0x4d0200 0xf000004 of:SLOF,git-3a259d hv:linux,kvm pSeries > > [ 8.857342] NIP: c008000000040a4c LR: c000000000ed25d0 CTR: > > c008000000040a4c > > [ 8.857413] REGS: c000000009b6f6f0 TRAP: 0400 Tainted: G W > > (6.6.37-un-def-alt1) > > [ 8.857510] MSR: 8000000010009033 <SF,EE,ME,IR,DR,RI,LE> CR: > > 28008286 XER: 00000000 > > [ 8.857588] CFAR: c000000000ed25cc IRQMASK: 0 > > [ 8.857588] GPR00: c000000000ed25a8 c000000009b6f990 > > c000000001ac7f00 c000000006130400 > > [ 8.857588] GPR04: c008000000920048 0000000000000001 > > 0000000000000000 0000000000000000 > > [ 8.857588] GPR08: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.857588] GPR12: c008000000040a4c c00000003fff7a00 > > 0000000000000000 0000000000000000 > > [ 8.857588] GPR16: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.857588] GPR20: 7fffffffffffffff 0000000000000000 > > 0000000000000001 0000000000000000 > > [ 8.857588] GPR24: c000000006130400 c000000006510a00 > > c000000006510f00 c0000000041a9000 > > [ 8.857588] GPR28: 0000000000000001 c000000006130400 > > 0000000000000000 c008000000920000 > > [ 8.858184] NIP [c008000000040a4c] bpf_prog_2fb4fda3a3499517+0x0/0x8c > > [ 8.858245] LR [c000000000ed25d0] sk_filter_trim_cap+0xc0/0x370 > > [ 8.858308] Call Trace: > > [ 8.858333] [c000000009b6f990] [c000000000ed2574] > > sk_filter_trim_cap+0x64/0x370 (unreliable) > > [ 8.858421] [c000000009b6fa10] [c000000001068b64] > > unix_dgram_sendmsg+0x214/0xb10 > > [ 8.858511] [c000000009b6fad0] [c000000000e4c59c] > > sock_write_iter+0x19c/0x1e0 > > [ 8.858586] [c000000009b6fb80] [c0000000005b1b58] vfs_write+0x258/0x4e0 > > [ 8.858648] [c000000009b6fc40] [c0000000005b21d4] ksys_write+0x114/0x170 > > [ 8.858711] [c000000009b6fc90] [c000000000030230] > > system_call_exception+0x190/0x390 > > [ 8.858785] [c000000009b6fe50] [c00000000000c7d4] > > system_call_common+0xf4/0x258 > > [ 8.858859] --- interrupt: c00 at 0x7fffb082b884 > > [ 8.858908] NIP: 00007fffb082b884 LR: 000000012a02ab70 CTR: > > 0000000000000000 > > [ 8.858979] REGS: c000000009b6fe80 TRAP: 0c00 Tainted: G W > > (6.6.37-un-def-alt1) > > [ 8.859060] MSR: 800000000280f033 > > <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28002281 XER: 00000000 > > [ 8.859153] IRQMASK: 0 > > [ 8.859153] GPR00: 0000000000000004 00007fffd85341f0 > > 00007fffb0936d00 0000000000000005 > > [ 8.859153] GPR04: 00007fffb068fffa 0000000000000006 > > 0000000000000001 0000000000000005 > > [ 8.859153] GPR08: 00007fffb068fffa 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.859153] GPR12: 0000000000000000 00007fffb09ea540 > > 0000000000000000 0000000000000000 > > [ 8.859153] GPR16: 0000000000000000 0000000000000000 > > 0000000000000000 0000000000000000 > > [ 8.859153] GPR20: 00007fffd85344b0 0000000000000000 > > 0000000000000001 0000000000000000 > > [ 8.859153] GPR24: 000000012a053698 000000000000008b > > 0000000000000000 0000000000000001 > > [ 8.859153] GPR28: 00007fffb068fffa 0000000000000005 > > 0000000000000006 000000012a053698 > > [ 8.859738] NIP [00007fffb082b884] 0x7fffb082b884 > > [ 8.859786] LR [000000012a02ab70] 0x12a02ab70 > > [ 8.859836] --- interrupt: c00 > > [ 8.859872] Code: 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 > > 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 7fe00008 > > <7fe00008> 7fe00008 7fe00008 7fe00008 > > [ 8.860013] ---[ end trace 0000000000000000 ]--- > > [ 8.863088] pstore: backend (nvram) writing error (-1) > > [ 8.863141] > > [ 8.863166] note: bpf_prog01[689] exited with irqs disabled > > > > And so on. Temporary build/test log is at > > https://git.altlinux.org/tasks/352218/build/100/ppc64le/log > > > > Other stable/longterm branches or other architectures does not exhibit this. > > > > Thanks, > > Hi all - this just took down a production server for me, on POWER9 bare > metal. Not running tests, just booting normally, before services even came > up. Had to perform manual restoration, reverting to 6.6.36 worked. Also > running 64k kernel, unsure if it's better on 4k kernel. > > In case it's helpful, here's the log from my boot: > https://dpaste.org/Gyxxg/raw Ok, this isn't good, something went wrong with my backports here. Let me go revert them all and push out a new 6.6.y release right away. thanks for the report! greg k-h
