On 2024/8/22 01:28, Ilya Leoshkevich wrote: > On Tue, 2024-08-20 at 17:38 -0700, Tony Ambardar wrote: > > > [...] > >> I used the command line: >> ./test_progs -d >> get_stack_raw_tp,stacktrace_build_id,verifier_iterating_callbacks,tai >> lcalls >> >> which includes the current DENYLIST.s390x as well as 'tailcalls', >> which >> is also excluded by the kernel-patches/bpf s390x CI. I note the CI >> excludes several more tests that seem to work. Any idea why that is? >> >> For reference, the issue with 'tailcalls/tailcall_hierarchy_count' is >> an >> RCU stall and kernel hang: >> >> root@(none):/usr/libexec/kselftests-bpf# ./test_progs -v --debug -n >> 332/19 >> bpf_testmod.ko is already unloaded. >> Loading bpf_testmod.ko... >> Successfully loaded bpf_testmod.ko. >> test_tailcall_hierarchy_count:PASS:load obj 0 nsec >> test_tailcall_hierarchy_count:PASS:find entry prog 0 nsec >> test_tailcall_hierarchy_count:PASS:prog_fd 0 nsec >> test_tailcall_hierarchy_count:PASS:find jmp_table 0 nsec >> test_tailcall_hierarchy_count:PASS:map_fd 0 nsec >> test_tailcall_hierarchy_count:PASS:update jmp_table 0 nsec >> test_tailcall_hierarchy_count:PASS:find data_map 0 nsec >> test_tailcall_hierarchy_count:PASS:open fentry_obj file 0 nsec >> test_tailcall_hierarchy_count:PASS:find fentry prog 0 nsec >> test_tailcall_hierarchy_count:PASS:set_attach_target subprog_tail 0 >> nsec >> test_tailcall_hierarchy_count:PASS:load fentry_obj 0 nsec >> test_tailcall_hierarchy_count:PASS:attach_trace 0 nsec >> rcu: INFO: rcu_sched self-detected stall on CPU >> rcu: 0-....: (1 GPs behind) idle=4eb4/1/0x4000000000000000 >> softirq=527/528 fqs=1050 >> rcu: (t=2100 jiffies g=-379 q=20 ncpus=2) >> CPU: 0 UID: 0 PID: 84 Comm: test_progs Tainted: G O >> 6.10.0-12706-g853081e84612-dirty #111 >> Tainted: [O]=OOT_MODULE >> Hardware name: QEMU 8561 QEMU (KVM/Linux) >> Krnl PSW : 0704f00180000000 000003ffe00f8fca >> (lock_release+0xf2/0x190) >> R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 RI:0 >> EA:3 >> Krnl GPRS: 00000000b298dd12 0000000000000000 000002f23fd767c8 >> 000003ffe1848800 >> 0000000000000001 0000037fe034edbc 0000037fe034fd74 >> 0000000000000001 >> 0700037fe034edc8 000003ffe0249e48 000003ffe1848800 >> 000003ffe19ba7c8 >> 000003ff9f7a7f90 0000037fe034ef00 000003ffe00f8f96 >> 0000037fe034ed78 >> Krnl Code: 000003ffe00f8fbe: a7820300 tmhh %r8,768 >> 000003ffe00f8fc2: a7840004 brc >> 8,000003ffe00f8fca >> #000003ffe00f8fc6: ad03f0a0 stosm 160(%r15),3 >> >000003ffe00f8fca: eb8ff0a80004 lmg >> %r8,%r15,168(%r15) >> 000003ffe00f8fd0: 07fe bcr 15,%r14 >> 000003ffe00f8fd2: c0e500011057 brasl >> %r14,000003ffe011b080 >> 000003ffe00f8fd8: ec26ffa6007e cij >> %r2,0,6,000003ffe00f8f24 >> 000003ffe00f8fde: c01000b78b96 larl >> %r1,000003ffe17ea70a >> Call Trace: >> [<000003ffe00f8fca>] lock_release+0xf2/0x190 >> ([<000003ffe00f8f96>] lock_release+0xbe/0x190) >> [<000003ffe0249ea4>] __bpf_prog_exit_recur+0x5c/0x68 >> [<000003ff6001e0b0>] bpf_trampoline_73014444060+0xb0/0xd2 >> [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8 >> [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8 >> [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8 >> [<000003ff60024d2a>] bpf_prog_eb7edc599e93dcc8_entry+0x72/0xc8 >> [<000003ff60024d2a>] bpf_prog_eb7edc599e93dcc8_entry+0x72/0xc8 >> [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8 >> [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8 >> [<000003ff60024d14>] bpf_prog_eb7edc599e93dcc8_entry+0x5c/0xc8 >> [<000003ffe084ecee>] bpf_test_run+0x216/0x3a8 >> [<000003ffe084f9cc>] bpf_prog_test_run_skb+0x21c/0x630 >> [<000003ffe0202ad2>] __sys_bpf+0x7ea/0xbb0 >> [<000003ffe0203114>] __s390x_sys_bpf+0x44/0 > > Thanks for the detailed analysis! I will need to port > > commit 116e04ba1459fc08f80cf27b8c9f9f188be0fcb2 > Author: Leon Hwang <hffilwlqm@xxxxxxxxx> > Date: Sun Jul 14 20:39:00 2024 +0800 > > bpf, x64: Fix tailcall hierarchy > > to s390x to fix this. > Hi Ilya, I think you should wait for a while, as I'll fix another tailcall issue in near future. After fixing the issue, you can port them both in one shot. Thanks, Leon
