On Sat, 2024-08-17 at 14:57 -0700, Tony Ambardar wrote: > Hello all, > > I'd appreciate some help from the BPF and s390x communities... > > Some background: I'm finalizing a patch series enabling full cross- > endian > support for libbpf and selftests/bpf, and testing with mips64 so far. > This > arch makes switching the build byte-order trivial and has been very > handy > for A/B testing, but lacks some basic support (bpf2bpf call, kfuncs, > etc.) > making for poor selftests coverage. > > To finish testing I thought (optimistically?) to cross-build kernel > and > bpf selftests from x86_64 to s390x. That configuration might also be > used > later on bpf-ci for regression testing endian support and sharing the > load > of s390x builds. > > Locally, the build succeeds but when running it under QEMU I see > kernel > crashes trying to load any modules (e.g. prng or bpf_testmod). > > > The build/test setup uses Ubuntu and gcc: > > kodidev:~/linux$ lsb_release -d > Description: Ubuntu 22.04.4 LTS > > kodidev:~/linux$ s390x-linux-gnu-gcc --version > s390x-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 > Copyright (C) 2021 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There > is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > PURPOSE. > > > The code targets bpf/master, and I've researched QEMU/s390x usage > details > online but mainly follow https://docs.kernel.org/bpf/s390.html. For > rootfs > I take the same s390x image used by the kernel-patches bpf-ci. > > > The kernel .config used is attached, and the QEMU command-line is > below: > > qemu-system-s390x -cpu max,zpci=on -smp 2 -nographic -m 1G \ > -object rng-random,filename=/dev/urandom,id=rng0 \ > -device virtio-rng-ccw,rng=rng0 \ > -device virtio-net-ccw,netdev=eth0 \ > -netdev user,id=eth0,hostfwd=tcp::2224-:22 \ > -serial mon:stdio \ > -nodefaults \ > -kernel bzImage-s390x \ > -drive file=root-s390x-glibc.qcow2,if=virtio,format=qcow2 \ > -append "rootwait root=/dev/vda rw net.ifnames=0 biosdevname=0 > console=ttyS1" > > > After successfully booting, the crashes are easily reproduced: > > root@(none):/# uname -a > Linux (none) 6.10.0-12706-g853081e84612-dirty #111 SMP Sat Aug 17 > 00:49:23 > PDT 2024 s390x GNU/Linux > > # modprobe prng > Unable to handle kernel pointer dereference in virtual kernel address > space > Failing address: 000003fee0011000 TEID: 000003fee0011803 > Fault in home space mode while using kernel ASCE. > AS:0000000001dac007 R3:0000000000000024 > Oops: 003b ilc:1 [#1] SMP > Modules linked in: prng(+) > CPU: 1 UID: 0 PID: 81 Comm: modprobe Not tainted 6.10.0-12691- > g52e8c345c9f0 #106 > Hardware name: QEMU 3906 QEMU (KVM/Linux) > Krnl PSW : 0704d00180000000 000003fee0011ea0 (0x3fee0011ea0) > R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 > EA:3 > Krnl GPRS: 000002f2820db180 000003ffe1ca49a8 0000000000000000 > 000003ff6000f498 > 0000000000000000 0000000000000a22 0000000000000000 > 0000000000000000 > 000003ffe18111e0 000002f2806a48e8 000003ff6000f498 > 0000000000000000 > 000003ff8eaacfa8 000002aa0bbafa00 000003ff6000f4be > 0000037fe0c33b58 > Krnl Code: Bad PSW. > Call Trace: > [<000003fee0011ea0>] 0x3fee0011ea0 > [<000003ffe0000b24>] do_one_initcall+0x64/0x258 > [<000003ffe01566e8>] do_init_module+0x78/0x258 > [<000003ffe0158160>] init_module_from_file+0x88/0xa8 > [<000003ffe01582e8>] idempotent_init_module+0x168/0x230 > [<000003ffe0158430>] __s390x_sys_finit_module+0x80/0xb8 > [<000003ffe0b6f58a>] __do_syscall+0x232/0x2b0 > [<000003ffe0b81b90>] system_call+0x70/0x98 > INFO: lockdep is turned off. > Last Breaking-Event-Address: > [<000003ff6000f4b8>] > cpu_feature_match_S390_CPU_FEATURE_MSA_init+0x20/0xb68 [prng] > Kernel panic - not syncing: Fatal exception: panic_on_oops > > > and also: > > # ./test_progs -a xdpwall > bpf_testmod: loading out-of-tree module taints kernel. > Unable to handle kernel pointer dereference in virtual kernel address > space > Failing address: 000003fee0293000 TEID: 000003fee0293803 > Fault in home space mode while using kernel ASCE. > AS:0000000001dac007 R3:0000000000000024 > Oops: 003b ilc:1 [#1] SMP > Modules linked in: bpf_testmod(O+) > CPU: 2 UID: 0 PID: 91 Comm: test_progs Tainted: G O > 6.10.0-12691-g52e8c345c9f0 #106 > Tainted: [O]=OOT_MODULE > Hardware name: QEMU 3906 QEMU (KVM/Linux) > Krnl PSW : 0704d00180000000 000003fee0293998 (0x3fee0293998) > R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 > EA:3 > Krnl GPRS: 000002f20374e760 0001677600016805 0000000000000000 > 000003ff6000e288 > 0000000000000000 000000000000007a 0000000000000000 > 0000000000000000 > 000003ffe18111e0 000002f204c6d2e8 000003ff6000ab90 > 0000000000000000 > 000003ffb73acfa8 0000000000000000 000003ff6000abce > 0000037fe0803b50 > Krnl Code: Bad PSW. > Call Trace: > [<000003fee0293998>] 0x3fee0293998 > [<000003ffe0000b24>] do_one_initcall+0x64/0x258 > [<000003ffe01566e8>] do_init_module+0x78/0x258 > [<000003ffe0158160>] init_module_from_file+0x88/0xa8 > [<000003ffe01582e8>] idempotent_init_module+0x168/0x230 > [<000003ffe0158430>] __s390x_sys_finit_module+0x80/0xb8 > [<000003ffe0b6f58a>] __do_syscall+0x232/0x2b0 > [<000003ffe0b81b90>] system_call+0x70/0x98 > INFO: lockdep is turned off. > Last Breaking-Event-Address: > [<000003ff6000abc8>] bpf_testmod_init+0x38/0x160 [bpf_testmod] > Kernel panic - not syncing: Fatal exception: panic_on_oops > > > Has anyone encountered this before, or is able to reproduce? > Could someone share a "known good" kernel .config working in the > past? > > I'd be grateful for any advice or helpful suggestions. > > Thanks, > Tony Hi, I assume you are using the distro qemu v6.2? Could you please try v9.1.0-rc2? It contains quite a few emulation bugfixes. Best regards, Ilya
