On Wed, Jan 31, 2024 at 10:47 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
>
> On 30/01/2024 23:05, Bryce Kahle wrote:
> > From: Bryce Kahle <bryce.kahle@xxxxxxxxxxxxx>
> >
> > Enables a user to generate minimized kernel module BTF.
> >
> > If an eBPF program probes a function within a kernel module or uses
> > types that come from a kernel module, split BTF is required. The split
> > module BTF contains only the BTF types that are unique to the module.
> > It will reference the base/vmlinux BTF types and always starts its type
> > IDs at X+1 where X is the largest type ID in the base BTF.
> >
> > Minimization allows a user to ship only the types necessary to do
> > relocations for the program(s) in the provided eBPF object file(s). A
> > minimized module BTF will still not contain vmlinux BTF types, so you
> > should always minimize the vmlinux file first, and then minimize the
> > kernel module file.
> >
> > Example:
> >
> > bpftool gen min_core_btf vmlinux.btf vm-min.btf prog.bpf.o
> > bpftool -B vm-min.btf gen min_core_btf mod.btf mod-min.btf prog.bpf.o
>
> This is great! I've been working on a somewhat related problem involving
> split BTF for modules, and I'm trying to figure out if there's overlap
> with what you've done here that can help in either direction. I'll try
> and describe what I'm doing. Sorry if this is a bit of a diversion,
> but I just want to check if there are potential ways your changes could
> facilitate other scenarios in the future.
>
> The problem I'm trying to tackle is to enable split BTF module
> generation to be more resilient to underlying kernel BTF changes;
> this would allow for example a module that is not built with the kernel
> to generate BTF and have it work even if small changes in vmlinux occur.
> Even a small change in BTF ids in base BTF is enough to invalidate the
> associated split BTF, so the question is how to make this a bit less
> brittle. This won't be needed for modules built along with the kernel,
> but more for cases like a package delivering a kernel module.
>
> The way this is done is similar to what you're doing - generating
> minimal base vmlinux BTF along with the module BTF. In my case however
> the minimization is not driven by CO-RE relocations; rather it is driven
> by only adding types that are referenced by module BTF and any other
> associated types needed. We end up with minimal base BTF that is carried
> along with the module BTF (in a .BTF.base_minimal section) and this
> minimal BTF will be used to later reconcile module BTF with the running
> kernel BTF when the module is loaded; it essentially provides the
> additional information needed to map to current vmlinux types.
>
> In this approach, minimal vmlinux BTF is generated via an additional
> option to pahole which adds an extra phase to BTF deduplication between
> module and kernel. Once we have found the candidate mappings for
> deduplication, we can look at all base BTF references from module BTF
> and recursively add associated types to the base minimal BTF. Finally we
> reparent the split BTF to this minimal base BTF. Experiments show most
> modules wind up with base minimal BTF of around 4000 types, so the
> minimization seems to work well. But it's complex.
>
> So what I've been trying to work out is if this dedup complexity can be
> eliminated with your changes, but from what I can see, the membership in
> the minimal base BTF in your case is driven by the CO-RE relocations
> used in the BPF program. Would there do you think be a future where we
> would look at doing base minimal BTF generation by other criteria (like
> references from the module BTF)? Thanks!
Hm... I might be misremembering or missing something, but the problem
you are solving doesn't seem to be related to BTF minimization. I also
forgot why you need BTF deduplication, I vaguely remember we needed to
remember "expectations" of types that module BTF references in vmlinux
BTF, but I fail to remember why we needed dedup... Perhaps we need a
BPF office hours session to go over details again?
>
> Alan
>
> > v3->v4:
> > - address style nit about start_id initialization
> > - rename base to src_base_btf (base_btf is a global var)
> > - copy src_base_btf so new BTF is not modifying original vmlinux BTF
> >
> > Signed-off-by: Bryce Kahle <bryce.kahle@xxxxxxxxxxxxx>
> > ---
> > .../bpf/bpftool/Documentation/bpftool-gen.rst | 18 ++++++++++-
> > tools/bpf/bpftool/gen.c | 32 +++++++++++++++----
> > 2 files changed, 42 insertions(+), 8 deletions(-)
> >
> > diff --git a/tools/bpf/bpftool/Documentation/bpftool-gen.rst b/tools/bpf/bpftool/Documentation/bpftool-gen.rst
> > index 5006e724d..e067d3b05 100644
> > --- a/tools/bpf/bpftool/Documentation/bpftool-gen.rst
> > +++ b/tools/bpf/bpftool/Documentation/bpftool-gen.rst
> > @@ -16,7 +16,7 @@ SYNOPSIS
> >
> > **bpftool** [*OPTIONS*] **gen** *COMMAND*
> >
> > - *OPTIONS* := { |COMMON_OPTIONS| | { **-L** | **--use-loader** } }
> > + *OPTIONS* := { |COMMON_OPTIONS| | { **-B** | **--base-btf** } | { **-L** | **--use-loader** } }
> >
> > *COMMAND* := { **object** | **skeleton** | **help** }
> >
> > @@ -202,6 +202,14 @@ OPTIONS
> > =======
> > .. include:: common_options.rst
> >
> > + -B, --base-btf *FILE*
> > + Pass a base BTF object. Base BTF objects are typically used
> > + with BTF objects for kernel modules. To avoid duplicating
> > + all kernel symbols required by modules, BTF objects for
> > + modules are "split", they are built incrementally on top of
> > + the kernel (vmlinux) BTF object. So the base BTF reference
> > + should usually point to the kernel BTF.
> > +
> > -L, --use-loader
> > For skeletons, generate a "light" skeleton (also known as "loader"
> > skeleton). A light skeleton contains a loader eBPF program. It does
> > @@ -444,3 +452,11 @@ ones given to min_core_btf.
> > obj = bpf_object__open_file("one.bpf.o", &opts);
> >
> > ...
> > +
> > +Kernel module BTF may also be minimized by using the -B option:
> > +
> > +**$ bpftool -B 5.4.0-smaller.btf gen min_core_btf 5.4.0-module.btf 5.4.0-module-smaller.btf one.bpf.o**
> > +
> > +A minimized module BTF will still not contain vmlinux BTF types, so you
> > +should always minimize the vmlinux file first, and then minimize the
> > +kernel module file.
> > diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c
> > index ee3ce2b80..57691f766 100644
> > --- a/tools/bpf/bpftool/gen.c
> > +++ b/tools/bpf/bpftool/gen.c
> > @@ -1630,6 +1630,7 @@ static int do_help(int argc, char **argv)
> > " %1$s %2$s help\n"
> > "\n"
> > " " HELP_SPEC_OPTIONS " |\n"
> > + " {-B|--base-btf} |\n"
> > " {-L|--use-loader} }\n"
> > "",
> > bin_name, "gen");
> > @@ -1695,14 +1696,14 @@ btfgen_new_info(const char *targ_btf_path)
> > if (!info)
> > return NULL;
> >
> > - info->src_btf = btf__parse(targ_btf_path, NULL);
> > + info->src_btf = btf__parse_split(targ_btf_path, base_btf);
> > if (!info->src_btf) {
> > err = -errno;
> > p_err("failed parsing '%s' BTF file: %s", targ_btf_path, strerror(errno));
> > goto err_out;
> > }
> >
> > - info->marked_btf = btf__parse(targ_btf_path, NULL);
> > + info->marked_btf = btf__parse_split(targ_btf_path, base_btf);
> > if (!info->marked_btf) {
> > err = -errno;
> > p_err("failed parsing '%s' BTF file: %s", targ_btf_path, strerror(errno));
> > @@ -2139,12 +2140,29 @@ static int btfgen_remap_id(__u32 *type_id, void *ctx)
> > /* Generate BTF from relocation information previously recorded */
> > static struct btf *btfgen_get_btf(struct btfgen_info *info)
> > {
> > - struct btf *btf_new = NULL;
> > + struct btf *btf_new = NULL, *src_base_btf_new = NULL;
> > unsigned int *ids = NULL;
> > + const struct btf *src_base_btf;
> > unsigned int i, n = btf__type_cnt(info->marked_btf);
> > - int err = 0;
> > + int start_id, err = 0;
> > +
> > + src_base_btf = btf__base_btf(info->src_btf);
> > + start_id = src_base_btf ? btf__type_cnt(src_base_btf) : 1;
> >
> > - btf_new = btf__new_empty();
> > + /* clone BTF to sanitize a copy and leave the original intact */
> > + if (src_base_btf) {
> > + const void *raw_data;
> > + __u32 sz;
> > +
> > + raw_data = btf__raw_data(src_base_btf, &sz);
> > + src_base_btf_new = btf__new(raw_data, sz);
> > + if (!src_base_btf_new) {
> > + err = -errno;
> > + goto err_out;
> > + }
> > + }
> > +
> > + btf_new = btf__new_empty_split(src_base_btf_new);
> > if (!btf_new) {
> > err = -errno;
> > goto err_out;
> > @@ -2157,7 +2175,7 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info)
> > }
> >
> > /* first pass: add all marked types to btf_new and add their new ids to the ids map */
> > - for (i = 1; i < n; i++) {
> > + for (i = start_id; i < n; i++) {
> > const struct btf_type *cloned_type, *type;
> > const char *name;
> > int new_id;
> > @@ -2213,7 +2231,7 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info)
> > }
> >
> > /* second pass: fix up type ids */
> > - for (i = 1; i < btf__type_cnt(btf_new); i++) {
> > + for (i = start_id; i < btf__type_cnt(btf_new); i++) {
> > struct btf_type *btf_type = (struct btf_type *) btf__type_by_id(btf_new, i);
> >
> > err = btf_type_visit_type_ids(btf_type, btfgen_remap_id, ids);
>
