Update struct btf_header to add a new "kind_layout" section containing a description of how to parse the BTF kinds known about at BTF encoding time. This provides the opportunity for tools that might not know all of these kinds - as is the case when older tools run on more newly-generated BTF - to still parse the BTF provided, even if it cannot all be used. Also add CRCs for the BTF and base BTF (if needed) from which it was created. CRCs provide a few useful features: - the base CRC allows us to explicitly identify when the split and base BTF are not matched - absence of a base BTF CRC can indicate that BTF is standalone; i.e. not defined relative to base BTF The former case can be used to explicitly reject mismatched module/kernel BTF rather than assuming it is matched until an unexpected type is encountered. The latter case is useful for modules that are not built as frequently as the kernel; in such cases, the module can be built standalone by specifying an empty BTF base: make BTF_BASE= M=path/2/module If CRCs are not present (as will be the case for pahole versions prior to the proposed v1.26 which will support CRC generation), standalone BTF can still be identified by a slower fallback method of examining BTF type ids to ensure that BTF is self-referential only. To ensure existing tooling can handle standalone BTF for kernel modules, we remap the type ids to start after the vmlinux BTF ids, to make it appear to be split BTF. This allows tools (and the kernel) that assume split BTF for modules to operate normally. Also add support to bpftool to dump metadata about BTF; its size, header information and kind layout section. The ideas here were discussed at [1], with further discussion at [2]. A patch for pahole [3] will enable the CRC/kind layout addition to BTF generation, but the kernel can still be built and tests added will still pass; it will just be the case that we fall back to the slowpath of standalone module identification in the absence of CRCs in the standalone module. Note that for additional context I will be discussing this work along with some other issues around evolving BTF at Linux Plumbers next week; see [4] where slides will be added shortly. Changes since v3 [5]: - fixed mismerge issues with kbuild changes for BTF generation (patches 9, 14) - fixed a few small issues in libbpf with kind layout representation (patches 2, 4) Changes since v2 [6]: - drop "optional" kind flag (Andrii, patch 1) - allocate "struct btf_header" for struct btf to ensure we can always access new fields (Andrii, patch 2) - use an internal BTF kind array in btf.c to simplify kind encoding (Andrii, patch 2) - drop use of kind layout information for in-kernel parsing, since the kernel needs to be strict in what it accepts (Andrii, patch 6) - added CRC verification for BTF objects and for matching with base object (Alexei, patches 7,8) - fixed bpftool json output (Quentin, patch 10) - added standalone module BTF support, tests (patches 13-17) Changes since RFC - Terminology change from meta -> kind_layout (Alexei and Andrii) - Simplify representation, removing meta header and just having kind layout section (Alexei) - Fixed bpftool to have JSON support, support prefix match, documented changes (Quentin) - Separated metadata opts into add_kind_layout and add_crc - Added additional positive/negative tests to cover basic unknown kind, one with an info_sz object following it and one with N elem_sz elements following it. - Updated pahole-flags to use help output rather than version to see if features are present [1] https://lore.kernel.org/bpf/CAEf4BzYjWHRdNNw4B=eOXOs_ONrDwrgX4bn=Nuc1g8JPFC34MA@xxxxxxxxxxxxxx/ [2] https://lore.kernel.org/bpf/20230531201936.1992188-1-alan.maguire@xxxxxxxxxx/ [3] https://lore.kernel.org/bpf/20231110111533.64608-1-alan.maguire@xxxxxxxxxx/ [4] https://lpc.events/event/17/contributions/1576/ [5] https://lore.kernel.org/bpf/20231110110304.63910-1-alan.maguire@xxxxxxxxxx/ [6] https://lore.kernel.org/bpf/20230616171728.530116-1-alan.maguire@xxxxxxxxxx/ Alan Maguire (17): btf: add kind layout encoding, crcs to UAPI libbpf: support kind layout section handling in BTF libbpf: use kind layout to compute an unknown kind size libbpf: add kind layout encoding, crc support libbpf: BTF validation can use kind layout for unknown kinds btf: support kernel parsing of BTF with kind layout bpf: add BTF CRC verification where present bpf: verify base BTF CRC to ensure it matches module BTF kbuild, bpf: switch to --btf_features, add crc,kind_layout features bpftool: add BTF dump "format meta" to dump header/metadata bpftool: update doc to describe bpftool btf dump .. format meta selftests/bpf: test kind encoding/decoding bpf: support standalone BTF in modules kbuild, bpf: allow opt-out from using split BTF for modules selftests/bpf: generalize module load to support specifying a module name selftests/bpf: build separate bpf_testmod module with standalone BTF selftests/bpf: update btf_module test to ensure standalone BTF works include/uapi/linux/btf.h | 18 + kernel/bpf/btf.c | 435 +++++++++++++- scripts/Makefile.btf | 5 + scripts/Makefile.modfinal | 4 +- .../bpf/bpftool/Documentation/bpftool-btf.rst | 30 +- tools/bpf/bpftool/bash-completion/bpftool | 2 +- tools/bpf/bpftool/btf.c | 91 ++- tools/include/uapi/linux/btf.h | 18 + tools/lib/bpf/btf.c | 341 ++++++++--- tools/lib/bpf/btf.h | 11 + tools/lib/bpf/libbpf.map | 1 + tools/testing/selftests/bpf/Makefile | 8 +- .../selftests/bpf/bpf_testmod/Makefile | 10 +- .../bpf_testmod_standalone-events.h | 57 ++ .../bpf/bpf_testmod/bpf_testmod_standalone.c | 551 ++++++++++++++++++ .../bpf/bpf_testmod/bpf_testmod_standalone.h | 31 + .../bpf_testmod_standalone_kfunc.h | 109 ++++ .../selftests/bpf/prog_tests/bpf_mod_race.c | 8 +- .../selftests/bpf/prog_tests/btf_kind.c | 176 ++++++ .../selftests/bpf/prog_tests/btf_module.c | 19 +- .../selftests/bpf/prog_tests/module_attach.c | 6 +- tools/testing/selftests/bpf/test_progs.c | 6 +- tools/testing/selftests/bpf/test_verifier.c | 6 +- tools/testing/selftests/bpf/testing_helpers.c | 24 +- tools/testing/selftests/bpf/testing_helpers.h | 4 +- 25 files changed, 1832 insertions(+), 139 deletions(-) create mode 100644 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_standalone-events.h create mode 100644 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_standalone.c create mode 100644 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_standalone.h create mode 100644 tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_standalone_kfunc.h create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_kind.c -- 2.31.1
