Hello,
For some time now I'm seeing multiple issues during BTF generation while
building recent kernels targeting mips64el, and would appreciate some help
to understand and fix the problems.
Some relate to resolve_btfids:
> LD vmlinux
> BTFIDS vmlinux
> WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> WARN: resolve_btfids: unresolved symbol bpf_key_put
> WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> NM System.map
> SORTTAB vmlinux
> OBJCOPY vmlinux.32
These do not appear to be #ifdef-related and have similar past reports [1].
I also see many pahole failures during BTF encoding of modules, such as:
> CC [M] net/ipv6/netfilter/nft_fib_ipv6.mod.o
> CC [M] net/ipv6/netfilter/ip6t_REJECT.mod.o
> CC [M] net/psample/psample.mod.o
> LD [M] crypto/cmac.ko
> BTF [M] crypto/cmac.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
> LD [M] lib/test_bpf.ko
> BTF [M] lib/test_bpf.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
> LD [M] lib/crc-ccitt.ko
> BTF [M] lib/crc-ccitt.ko
> die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> die__process_unit: tag not supported 0x11 (compile_unit)!
> die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> LD [M] lib/libcrc32c.ko
> BTF [M] lib/libcrc32c.ko
> die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> die__process_unit: tag not supported 0x11 (compile_unit)!
> die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> LD [M] lib/ts_kmp.ko
> BTF [M] lib/ts_kmp.ko
> LD [M] lib/ts_bm.ko
> BTF [M] lib/ts_bm.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
I have seen reports of various similar "die__" messages on the dwarves
list and repo, with the hint of an elfutils connection [2] but nothing
conclusive.
Details of the git commit and build environment are as follows:
> $ git log -1 --oneline bpf/master
> 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> test_sk_storage_tracing.c
>
> $ lsb_release -a
> Description: Ubuntu 22.04.4 LTS
>
> $ cat gcc-compile.txt
> ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
>
> $ mips64el-linux-gnuabi64-gcc --version
> mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
>
> $ mips64el-linux-gnuabi64-ld --version
> GNU ld (GNU Binutils for Ubuntu) 2.38
>
> $ pahole --version
> v1.26
>
> $ ldd $(which pahole)
> linux-vdso.so.1 (0x00007fff16f3f000)
> libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
> libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
> liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
> libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
>
> $ dpkg -s elfutils
> Package: elfutils
> ...
> Version: 0.186-1build1
> Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
For reference, I also attached the full .config and build log from the
above.
I should add this is not only a problem with the latest bpf/master but
also appears to affect the 6.6.x LTS kernel, which I tested while building
a mips64el OpenWrt distro image. That build environment employs the latest
gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
Not only do I see similar warnings from resolve_btfids and pahole, but
while running the distro image I encounter module loading failures that
suggest ELF corruption in some module .ko files, based on the following:
> root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> ...
> init_module(0xfff3e36160, 307448, "") = -1 EINVAL (Invalid argument)
> ...
> $ man init_module
> ...
> The following errors may additionally occur for init_module():
> ...
> EINVAL param_values is invalid, or some part of the ELF image in
> module_image contains inconsistencies.
> ...
I'd be grateful if some of the BTF/pahole experts could please review this
issue and share next steps or other details I might provide.
Thanks,
Tony Ambardar
Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@xxxxxxxxx/ [1]
Link: https://github.com/acmel/dwarves/issues/45 [2]
Attachment:
.config.gz
Description: application/gzip
Attachment:
mips64el-linux-gnuabi64.log.gz
Description: application/gzip
