On Thu, Jan 6, 2022 at 8:53 AM Buchberger, Dennis <dennis.buchberger@xxxxxxxxxxxxxxxx> wrote: > > > > [...] > > > As the target kernel does not support CONFIG_DEBUG_INFO_BTF, I used pahole -J (v1.22) to create vmlinux file with BTF info embedded there. > > > Basically, I followed this mails: > > > https://urldefense.com/v3/__https://lore.kernel.org/bpf/CADmGQ*1euj7Uv > > > 9e8UyZMMXDiYAKqXe9=GSTBFNbbg1E0R-ejyg@xxxxxxxxxxxxxx/__;Kw!!MeVeBiz6!5 > > > PZT7QBM-W93AhbZnRJ3kmTO4JyBUiapxeJrPCl4k4gKHidI5Ri0WQp19MbBDFP1nWOL3A$ > > > > > > Right now, the bpf program is just a uProbe for a simple test app, which writes some output to the tracing pipe. As Kernel 4.18. does not support global data for bpf programs, I had to remove (comment out) the bpf_trace_printk statements. > > > > You can do #define BPF_NO_GLOBAL_DATA before including bpf_helpers.h and then you can still use bpf_printk() helper macro. > > > > Thank you! That's really helpful. Is there any collection of up-to-date documentation and best practices in writing bpf code (without skeletons) besides the sources in libbpf/libbpf-bootstrap, iovisor /libbpf-tools and linux/tools/testing/selftests/bpf ? There are also various blog posts on the Internet. I tried to do my part with CO-RE at [0], but I don't think there is any single documentation that captures all possible issues in one place (and I don't think it's possible, there are just way too many variables to anticipate all possible problems). [0] https://nakryiko.com/posts/bpf-core-reference-guide/ > > > > > > > On the development machine, it works fine. But on the target machine, loading the program fails: libbpf: load bpf program failed: Invalid argument (full libbpf log see below). > > > When compiling the programs on the target machine without using CO:RE, I get a similar error (invalid argument, -22). > > > What could be the problem? I don't think the eBPF program uses anything that is available on Kernel 5.4.0 and not available on the system with Kernel 4.18, does it? > > > > > > Thanks in advance for your help. > > > Best > > > Dennis > > > > > > > > > > > > > > > ============ log ============ > > > > > > sudo ./ebpf > > > libbpf: loading main.bpf.o > > > [...] > > > libbpf: CO-RE relocating [0] struct pt_regs: found target candidate > > > [201] struct pt_regs in [vmlinux] > > > libbpf: prog 'trace_func_entry': relo #0: kind <byte_off> (0), spec is > > > [2] struct pt_regs.di (0:14 @ offset 112) > > > libbpf: prog 'trace_func_entry': relo #0: matching candidate #0 [201] > > > struct pt_regs.di (0:14 @ offset 112) > > > libbpf: prog 'trace_func_entry': relo #0: patched insn #2 (ALU/ALU64) > > > imm 112 -> 112 > > > libbpf: prog 'trace_func_entry': relo #1: kind <byte_off> (0), spec is > > > [2] struct pt_regs.si (0:13 @ offset 104) > > > libbpf: prog 'trace_func_entry': relo #1: matching candidate #0 [201] > > > struct pt_regs.si (0:13 @ offset 104) > > > libbpf: prog 'trace_func_entry': relo #1: patched insn #9 (ALU/ALU64) > > > imm 104 -> 104 > > > libbpf: sec 'kretprobe/': found 1 CO-RE relocations > > > libbpf: prog 'trace_func_exit': relo #0: kind <byte_off> (0), spec is > > > [2] struct pt_regs.ax (0:10 @ offset 80) > > > libbpf: prog 'trace_func_exit': relo #0: matching candidate #0 [201] > > > struct pt_regs.ax (0:10 @ offset 80) > > > libbpf: prog 'trace_func_exit': relo #0: patched insn #2 (ALU/ALU64) > > > imm 80 -> 80 > > > > CO-RE relocations succeeded, btf_custom_path worked, the problem is not in CO-RE. > > > > > libbpf: load bpf program failed: Invalid argument > > > libbpf: failed to load program 'trace_func_entry' > > > > I suspect this is due to your target machine running Ubuntu 18.10. > > Ubuntu has infamous problem with reporting kernel version through > > uname() syscall. I've just improved libbpf's detection of it few days ago (see [0]), but it didn't yet make it into Github mirror of libbpf. > > I'm going to start the sync right now, but you can manually specify correct version code with bpf_object__set_kversion() as you already realized. See how I do that in my patch [0], you can do that manually as well. > > > > [0] https://urldefense.com/v3/__https://patchwork.kernel.org/project/netdevbpf/patch/20211222231003.2334940-1-andrii@xxxxxxxxxx/__;!!MeVeBiz6!5PZT7QBM-W93AhbZnRJ3kmTO4JyBUiapxeJrPCl4k4gKHidI5Ri0WQp19MbBDFM_zuNNKA$ > > That's it, thank you very much! > > I just added > __u32 currKernelVersion = get_kernel_version(); > bpf_object__set_kversion(main_bpf_obj, currKernelVersion) > with get_kernel_version() from libbpf.c and your patch [0] and now it is working. :) > Great, once [1] is merged (I'll need to fix up selftests first), libbpf will be able to deal with this Ubuntu quirk automatically and you'll never know it existed. [1] https://github.com/libbpf/libbpf/pull/435 > However, there is something I do not understand: > (a): Why did it work on the development machine in the first place with Ubuntu 20.04? > - the old get_kernel_version() returned 328704 (5.4.0), the new version returns 328852 (5.4.148) so there already is a missmatch. (and I did not call it, so libbpf set the kernel version to 0?) You dev machine has recent enough kernel that doesn't check kernel version anymore, so it doesn't matter that Ubuntu reports the wrong one. > - on the target machine, it is 266752 (4.2.0) vs. 266772 (4.2.4) > (b): I use a uProbe. Why is the bpf program type kProbe? Is it just for usability of libbpf as for a uProbe the user must specify the address while for a kprobe only the symbol is required? 4.2 is old enough where kernel version has to match and it didn't. > (c): Why does the kernel version matter at all? > > It would be really nice if you could answer these questions as well or point me to a source. > > Best > Dennis > > [...] >
