On Tue, Oct 31, 2023 at 8:45 AM Lorenz Bauer <lorenz.bauer@xxxxxxxxxxxxx> wrote: > > I think there is something weird going on with BTF_TYPE_ID_LOCAL. A > call to bpf_obj_new is getting the "wrong" type as an argument as > emitted by clang into the instruction stream. > > Compiling local_kptr_stash.c from 6.6 selftest with clang-16 and > clang-17 yields the following disassembly for the stash_plain function > (source is at [1]): > > 00000000000001b0 <stash_plain>: > ... > 64: 18 01 00 00 1c 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x1c ll > 66: b7 02 00 00 00 00 00 00 r2 = 0x0 > 67: 85 10 00 00 ff ff ff ff call -0x1 > ; bpf_obj_new > ... > > This is looking at local_kptr_stash.bpf.linked3.o specifically, but > local_kptr_stash.bpf.o has the same problem. > > 0x1c being 28. From bpftool we can see that 28 corresponds to a FUNC > type which doesn't make much sense to me: > > [28] FUNC 'unstash_rb_node' type_id=15 linkage=global > > The source code actually passes struct plain_local to bpf_obj_new, > which has type ID 27: > > [27] STRUCT 'plain_local' size=16 vlen=2 > 'key' type_id=16 bits_offset=0 > 'data' type_id=16 bits_offset=64 > > I'm guessing that this works in practice since the CO-RE relo in > ext_infos actually carries the correct local_type_id: > > CORERelocation(local_type_id, Struct:"node_data"[0], local_id=18) > CORERelocation(local_type_id, Struct:"node_data"[0], local_id=18) > CORERelocation(local_type_id, Struct:"plain_local"[0], local_id=27) > > 1: https://elixir.bootlin.com/linux/v6.6/source/tools/testing/selftests/bpf/progs/local_kptr_stash.c#L76 I don't remember if this is intention or not, but the main part is adjusting CO-RE relocation, the actual instruction value is less important. But this is happening after static linking, because BTF is deduplicated (there is a duplication in BTF generated by Clang). $ llvm-objdump -dr ./local_kptr_stash.bpf.linked3.o | grep -A1 '65:' 65: 18 01 00 00 1c 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x1c ll 0000000000000208: CO-RE <local_type_id> [27] struct plain_local $ bpftool btf dump file ./local_kptr_stash.bpf.linked3.o | grep 'plain_local' [27] STRUCT 'plain_local' size=16 vlen=2 $ llvm-objdump -dr ./local_kptr_stash.bpf.o | grep -A1 '65:' 65: 18 01 00 00 1c 00 00 00 00 00 00 00 00 00 00 00 r1 = 0x1c ll 0000000000000208: CO-RE <local_type_id> [28] struct plain_local $ bpftool btf dump file ./local_kptr_stash.bpf.o | grep 'plain_local' [28] STRUCT 'plain_local' size=16 vlen=2 Note how plain_local was [28] and then became [27]. CO-RE relocation was correctly adjusted, but the instruction itself was left intact. The BTF duplication is in [26] FUNC_PROTO '(anon)' ret_type_id=16 vlen=1 'ctx' type_id=14 There are at least two identical prototypes (which is strange and might be worth looking into from Clang side).
