On Tue, Aug 20, 2024 at 05:38:49PM -0700, Tony Ambardar wrote: > > Hi Ilya, > > Thanks for following up. As it happens, I did this the day before out of > desperation after trying various kernel config and rootfs changes > with no luck, and can confirm the system runs faster and without the > kernel crashes noted above. Certainly the latest QEMU seems mandatory. > > The good news is that 99% of tests with my cross-compiled test_progs > work as expected out of the box, and some of the failing ones helped > troubleshoot a few hidden libbpf issues. I'll outline the remaining > failures for your feedback and comparison with native-built tests. > > I used the command line: > ./test_progs -d get_stack_raw_tp,stacktrace_build_id,verifier_iterating_callbacks,tailcalls > [snip] > Aside from the tests above, I see only 3 failing tests: > > All error logs: > test_map_ptr:PASS:skel_open 0 nsec > test_map_ptr:FAIL:skel_load unexpected error: -22 (errno 22) > #165 map_ptr:FAIL > subtest_userns:PASS:socketpair 0 nsec > subtest_userns:PASS:fork 0 nsec > recvfd:PASS:recvmsg 0 nsec > recvfd:PASS:cmsg_null 0 nsec > recvfd:PASS:cmsg_len 0 nsec > recvfd:PASS:cmsg_level 0 nsec > recvfd:PASS:cmsg_type 0 nsec > parent:PASS:recv_bpffs_fd 0 nsec > materialize_bpffs_fd:PASS:fs_cfg_cmds 0 nsec > materialize_bpffs_fd:PASS:fs_cfg_maps 0 nsec > materialize_bpffs_fd:PASS:fs_cfg_progs 0 nsec > materialize_bpffs_fd:PASS:fs_cfg_attachs 0 nsec > parent:PASS:materialize_bpffs_fd 0 nsec > sendfd:PASS:sendmsg 0 nsec > parent:PASS:send_mnt_fd 0 nsec > recvfd:PASS:recvmsg 0 nsec > recvfd:PASS:cmsg_null 0 nsec > recvfd:PASS:cmsg_len 0 nsec > recvfd:PASS:cmsg_level 0 nsec > recvfd:PASS:cmsg_type 0 nsec > parent:PASS:recv_token_fd 0 nsec > parent:FAIL:waitpid_child unexpected error: 22 (errno 3) > #402/9 token/obj_priv_implicit_token_envvar:FAIL > #402 token:FAIL > libbpf: prog 'on_event': BPF program load failed: Bad address > libbpf: prog 'on_event': -- BEGIN PROG LOAD LOG -- > The sequence of 8193 jumps is too complex. > verification time 2816240 usec > stack depth 360 > processed 116096 insns (limit 1000000) max_states_per_insn 1 total_states 5061 peak_states 5061 mark_read 2540 > -- END PROG LOAD LOG -- > libbpf: prog 'on_event': failed to load: -14 > libbpf: failed to load object 'pyperf600.bpf.o' > scale_test:FAIL:expect_success unexpected error: -14 (errno 14) > #525 verif_scale_pyperf600:FAIL > Summary: 559/4166 PASSED, 98 SKIPPED, 3 FAILED > Hi Ilya, A brief update with some good news: the 3 test failures above have been resolved and all expected tests now pass on QEMU/s390x under x86_64. Test '#165 map_ptr:FAIL' was a bug in my light-skeleton code, and fixed in my patch series v2: https://lore.kernel.org/bpf/cover.1724313164.git.tony.ambardar@xxxxxxxxx/ Test '#402/9 token/obj_priv_implicit_token_envvar:FAIL' was a problem in my rootfs configuration and now passes after resolving. Test '#525 verif_scale_pyperf600:FAIL' was caused by clang miscompilation exposed by my use of clang-19 and clang-20. The test passes when built with clang-17 (used by BPF CI) or clang-18 which I switched to use. One symptom of the problem is easily seen by manually compiling: $ clang-18 -g -Wall -Werror -D__TARGET_ARCH_s390 -mbig-endian -Itools/testing/selftests/bpf/tools/include -Itools/testing/selftests/bpf -Itools/include/uapi -Itools/testing/selftests/usr/include -Wno-compare-distinct-pointer-types -idirafter /usr/lib/llvm-18/lib/clang/18/include -idirafter /usr/local/include -idirafter /usr/lib/gcc-cross/s390x-linux-gnu/11/../../../../s390x-linux-gnu/include -idirafter /usr/include/s390x-linux-gnu -idirafter /usr/include -DENABLE_ATOMICS_TESTS -O2 --target=bpfeb -c tools/testing/selftests/bpf/progs/pyperf600.c -mcpu=v3 -o pyperf600.clang18.bpf.o $ clang-19 -g -Wall -Werror -D__TARGET_ARCH_s390 -mbig-endian -Itools/testing/selftests/bpf/tools/include -Itools/testing/selftests/bpf -Itools/include/uapi -Itools/testing/selftests/usr/include -Wno-compare-distinct-pointer-types -idirafter /usr/lib/llvm-19/lib/clang/19/include -idirafter /usr/local/include -idirafter /usr/lib/gcc-cross/s390x-linux-gnu/11/../../../../s390x-linux-gnu/include -idirafter /usr/include/s390x-linux-gnu -idirafter /usr/include -DENABLE_ATOMICS_TESTS -O2 --target=bpfeb -c tools/testing/selftests/bpf/progs/pyperf600.c -mcpu=v3 -o pyperf600.clang19.bpf.o $ llvm-readelf-18 -S pyperf600.clang{18,19}.bpf.o |grep .symtab [27] .symtab SYMTAB 0000000000000000 1739d0 01ad60 18 1 4572 8 [27] .symtab SYMTAB 0000000000000000 14f048 0001e0 18 1 12 8 Notice that the .symtab has shrunk by ~200X for example going to clang-19! (CCing llvm maintainers) Kind regards, Tony
