On Fri, Sep 13, 2024 at 1:59 AM Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote: > > On Thu, 12 Sep 2024 18:55:40 -0700 > Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx> wrote: > > > On Thu, Sep 12, 2024 at 4:54 PM Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote: > > > > > > On Thu, 12 Sep 2024 11:41:17 -0700 > > > Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx> wrote: > > > > > > > + BPF ML > > > > > > > > On Thu, Sep 12, 2024 at 8:35 AM <bot+bpf-ci@xxxxxxxxxx> wrote: > > > > > > > > > > Dear patch submitter, > > > > > > > > > > CI has tested the following submission: > > > > > Status: FAILURE > > > > > Name: [v14,00/19] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph > > > > > Patchwork: https://patchwork.kernel.org/project/netdevbpf/list/?series=889822&state=* > > > > > Matrix: https://github.com/kernel-patches/bpf/actions/runs/10833792984 > > > > > > > > > > Failed jobs: > > > > > test_progs-aarch64-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061791397 > > > > > test_progs_no_alu32-aarch64-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061791836 > > > > > test_progs-s390x-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061757062 > > > > > test_progs_no_alu32-s390x-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061757809 > > > > > > > > > > First test_progs failure (test_progs-aarch64-gcc): > > > > > #132 kprobe_multi_testmod_test > > > > > serial_test_kprobe_multi_testmod_test:PASS:load_kallsyms_local 0 nsec > > > > > #132/1 kprobe_multi_testmod_test/testmod_attach_api_syms > > > > > test_testmod_attach_api:PASS:fentry_raw_skel_load 0 nsec > > > > > trigger_module_test_read:PASS:testmod_file_open 0 nsec > > > > > test_testmod_attach_api:PASS:trigger_read 0 nsec > > > > > kprobe_multi_testmod_check:FAIL:kprobe_test1_result unexpected kprobe_test1_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kprobe_test2_result unexpected kprobe_test2_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kprobe_test3_result unexpected kprobe_test3_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kretprobe_test1_result unexpected kretprobe_test1_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kretprobe_test2_result unexpected kretprobe_test2_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kretprobe_test3_result unexpected kretprobe_test3_result: actual 0 != expected 1 > > > > > #132/2 kprobe_multi_testmod_test/testmod_attach_api_addrs > > > > > test_testmod_attach_api_addrs:PASS:ksym_get_addr_local 0 nsec > > > > > test_testmod_attach_api_addrs:PASS:ksym_get_addr_local 0 nsec > > > > > test_testmod_attach_api_addrs:PASS:ksym_get_addr_local 0 nsec > > > > > test_testmod_attach_api:PASS:fentry_raw_skel_load 0 nsec > > > > > trigger_module_test_read:PASS:testmod_file_open 0 nsec > > > > > test_testmod_attach_api:PASS:trigger_read 0 nsec > > > > > kprobe_multi_testmod_check:FAIL:kprobe_test1_result unexpected kprobe_test1_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kprobe_test2_result unexpected kprobe_test2_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kprobe_test3_result unexpected kprobe_test3_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kretprobe_test1_result unexpected kretprobe_test1_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kretprobe_test2_result unexpected kretprobe_test2_result: actual 0 != expected 1 > > > > > kprobe_multi_testmod_check:FAIL:kretprobe_test3_result unexpected kretprobe_test3_result: actual 0 != expected 1 > > > > > > > > > > > > > Seems like this selftest is still broken. Please let me know if you > > > > need help with building and running BPF selftests to reproduce this > > > > locally. > > > > > > Thanks, It will be helpful. Also I would like to know, is there any > > > debug mode (like print more debug logs)? > > > > So first of all, the assumption is that you will build a most recent > > kernel with Kconfig values set from > > tools/testing/selftests/bpf/config, so make sure you append that to > > your kernel config. Then build the kernel, BPF selftests' Makefile > > tries to find latest built kernel (according to KBUILD_OUTPUT/O/etc). > > > > Now to building BPF selftests: > > > > $ cd tools/testing/selftests/bpf > > $ make -j$(nproc) > > > > You'll need decently recent Clang and a few dependent packages. At > > least elfutils-devel, libzstd-devel, but there might be a few more > > which I never can remember. > > > > Once everything is built, you can run the failing test with > > > > $ sudo ./test_progs -t kprobe_multi_testmod_test -v > > > > The source code for user space part for that test is in > > prog_tests/kprobe_multi_testmod_test.c and BPF-side is in > > progs/kprobe_multi.c. > > Thanks for the information! > > > > > Taking failing output from the test: > > > > > > > kprobe_multi_testmod_check:FAIL:kretprobe_test3_result unexpected kretprobe_test3_result: actual 0 != expected 1 > > > > kretprobe_test3_result is a sort of identifier for a test condition, > > you can find a corresponding line in user space .c file grepping for > > that: > > > > ASSERT_EQ(skel->bss->kretprobe_testmod_test3_result, 1, > > "kretprobe_test3_result"); > > > > Most probably the problem is in: > > > > __u64 addr = bpf_get_func_ip(ctx); > > Yeah, and as I replyed to another thread, the problem is > that the ftrace entry_ip is not symbol ip. > > We have ftrace_call_adjust() arch function for reverse > direction (symbol ip to ftrace entry ip) but what we need > here is the reverse translate function (ftrace entry to symbol) > > The easiest way is to use kallsyms to find it, but this is > a bit costful (but it just increase bsearch in several levels). > Other possible options are > > - Change bpf_kprobe_multi_addrs_cmp() to accept a range > of address. [sym_addr, sym_addr + offset) returns true, > bpf_kprobe_multi_cookie() can find the entry address. > The offset depends on arch, but 16 is enough. This feels quite sloppy, tbh... > > - Change bpf_kprobe_multi_addrs_cmp() to call > ftrace_call_adjust() before comparing. This may take a > cost but find actual entry address. too expensive, unnecessary runtime overhead > > - (Cached method) when making link->addrs, make a link->adj_addrs > array too, where adj_addrs[i] == ftrace_call_adjust(addrs[i]). > > Let me try the 3rd one. It may consume more memory but the > fastest solution. I like the third one as well, but I'm not following why we'd need more memory. We can store adjusted addresses in existing link->addrs, and we'll need to adjust them to originals (potentially expensively) only in bpf_kprobe_multi_link_fill_link_info(). But that's not a hot path, so it doesn't matter. > > Thank you, > > > > > which is then checked as > > > > if ((const void *) addr == &bpf_testmod_fentry_test3) > > > > > > With your patches this doesn't match anymore. > > It actually enables the fprobe on those architectures, so > without my series, those test should be skipped. Ok, cool, still, let's fix the issue. > > Thank you, > > > > > > > Hopefully the above gives you some pointers, let me know if you run > > into any problems. > > > > > > > > Thank you, > > > > > > > > > > > > > > > > > Please note: this email is coming from an unmonitored mailbox. If you have > > > > > questions or feedback, please reach out to the Meta Kernel CI team at > > > > > kernel-ci@xxxxxxxx. > > > > > > > > > -- > > > Masami Hiramatsu (Google) <mhiramat@xxxxxxxxxx> > > > -- > Masami Hiramatsu (Google) <mhiramat@xxxxxxxxxx>
