On 01/08/2023 02:01, Masami Hiramatsu (Google) wrote: > On Mon, 31 Jul 2023 16:45:24 +0100 > Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: > >> On 27/07/2023 02:51, Masami Hiramatsu (Google) wrote: >>> On Thu, 27 Jul 2023 09:38:14 +0900 >>> Masami Hiramatsu (Google) <mhiramat@xxxxxxxxxx> wrote: >>> >>>>> Yep, BPF generation is more selective about what it emits in 1.25 to >>>>> avoid cases where a kernel function signature is ambiguous (multiple >>>>> functions of the same name with different signatures) or where it has >>>>> unexpected register use. You can observe this via pahole's --verbose >>>>> option (a lot of outut is emitted): >>>>> >>>>> In a built kernel directory (where unstripped vmlinux is present): >>>>> $ PAHOLE_FLAGS=$(./scripts/pahole_flags) >>>>> $ PAHOLE=/usr/local/bin/pahole >>>>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out >>>> >>>> So this will generate BTF from vmlinux DWARF info. >>>> >>>>> If you want to investigate why a function has been left out, look for >>>>> "skipping" verbose output like this: >>>>> >>>>> skipping addition of 'access_error'(access_error) due to multiple >>>>> inconsistent function prototypes >>>>> skipping addition of >>>>> 'acpi_ex_convert_to_object_type_string'(acpi_ex_convert_to_object_type_string.isra.0) >>>>> due to unexpected register used for parameter >>>> >>>> Ah, that's nice. Let me try. >>> >>> $ pahole --version >>> v1.23 >>> >> >> shouldn't this be v1.25? Is it possible pahole is picking up the wrong >> libdwarves? what does "ldd pahole" say? > > Here it is. > $ ldd pahole > linux-vdso.so.1 (0x00007ffd6b1e2000) > libdwarves_reorganize.so.1 => /opt/local/pahole/libdwarves_reorganize.so.1 (0x00007f1ddaad9000) > libdwarves.so.1 => /opt/local/pahole/libdwarves.so.1 (0x00007f1ddaa72000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1dda82a000) > libdw.so.1 => /usr/local/lib/x86_64-linux-gnu/libdw.so.1 (0x00007f1dda78c000) > libelf.so.1 => /usr/local/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f1dda771000) > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1dda753000) > /lib64/ld-linux-x86-64.so.2 (0x00007f1ddaaef000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1dda74e000) > liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1dda723000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1dda71e000) > > Maybe libdwarvers is not related, I could build pahole v1.23 and v1.22 without > sync the submodule. > > I also confirmed the same issue on Ubuntu 22.04's combination, which > update pahole from v1.22 to v1.25 recently. (but gcc is still v11.3) > > gcc-11.3 | clang-16.0 > ---------------------------+---------------- > v1.22 OK OK > (ubuntu) > v1.22 OK OK > (local) > v1.23 NG OK > (local) > v1.24 NG - > (local) > v1.25 NG - > (ubuntu) > v1.25 NG OK > (local) > > So, as far as I checked, there is something wrong between v1.22 and v1.23 > which is also related to gcc-11.3. > One thing that is notable about gcc 11 is that I believe it's the first gcc release to emit DWARF5 by default. I wonder if it's possible that the kernel emitted DWARF5, but pahole was built with libraries that didn't support it yet? Not sure how that fits with the fact that pahole v1.22 works though. If you have a chance, it might be worth experimenting with your kernel .config to specify DWARF4 to see if that makes a difference. You can check DWARF version associated with CUs with readelf --debug-dump=info vmlinux |grep -A 2 'Compilation Unit' >> >>> $ pahole --verbose -J $PAHOLE_FLAGS vmlinux > /tmp/pahole.out >>> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got INVALID (0x0)! >>> >>> OK, so something failed. >>> >>> $ grep skipping /tmp/pahole.out | wc -l >>> 0 >>> >>> Nothing to be skipped. >>> >>> $ grep -w kfree /tmp/pahole.out | wc -l >>> 0 >>> $ grep -w vfs_read /tmp/pahole.out | wc -l >>> 0 >>> >>> But both kfree and vfs_read are not found. >>> >>> $ perf probe -k ./vmlinux -V kfree >>> Available variables at kfree >>> @<kfree+0> >>> (unknown_type) object >>> $ perf probe -k ./vmlinux -V vfs_read >>> Available variables at vfs_read >>> @<vfs_read+0> >>> char* buf >>> loff_t* pos >>> size_t count >>> struct file* file >>> >>> However, perf probe can find both in the DWARF info. >>> >>> Thank you, >>> >> >> Unfortunately (or fortunately?) I haven't been able to reproduce so far >> I'm afraid. I used your config and built gcc 13 from source; everything >> worked as expected, with no warnings or missing functions (aside from >> the ones skipped due to inconsistent prototypes etc). > > Yeah, so I think gcc-11.3 is suspicious too (and it seems fixed in gcc-13). > > >> One other thing I can think of - is it possible libdw[arf]/libelf >> versions might be having an effect here? I'm using libdwarf.so.1.2, >> libdw-0.188, libelf-0.188. I can try and match yours. Thanks! > > Both libdw/libelf are 0.181. I didn't install libdwarf. > Hmm, I should update the libdw (elfutils) too. That might help. Thanks! Alan > Thank you, > >> >> Alan > >
