Re: [PATCH dwarves 0/7] Add support for generating BTF for all variables

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Sep 9, 2022 at 12:31 PM Stephen Brennan
<stephen.s.brennan@xxxxxxxxxx> wrote:
>
> >> (a) While we save space on vmlinux BTF, each module will have a bit of
> >>     extra data for variable types. On my laptop (5.15 based) I have 9.8
> >>     MB of BTF, and if you deduct vmlinux, you're still left with 4.7 MB.
> >>     If we assume the same overhead of 23.7%, that would be 1.1 MB of
> >>     extra module BTF for my particular use case.
> >>
> >>     $ ls -l /sys/kernel/btf | awk '{sum += $5} END {print(sum)}'
> >>     9876871
> >>     $ ls -l /sys/kernel/btf/vmlinux
> >>     -r--r--r-- 1 root root 5174406 Sep  7 14:20 /sys/kernel/btf/vmlinux
> >>
> >> (b) It's possible for "vmlinux-btf-extras" and "$MODULE" to contain
> >>     duplicate type definitions, wasting additional space. However, as
> >>     far as I understand it, this was already a possibility, e.g.
> >>     $MODULE1 and $MODULE2 could already contain duplicate types. So I
> >>     think this downside is no more.
> >
> > Both concerns are valid, but I'm a bit puzzled with (a).
> > At least in the networking drivers the number of global vars is very small.
> > I expected other drivers to be similar.
> > So having "functions and all vars" in ko-s should not add
> > that much overhead.
> >
> > Maybe you're seeing this overhead because pahole is adding
> > all declared vars and not only the vars that are actually present?
> > That would explain the discrepancy.
> > (b) with a bunch of duplicates is a sign that something is off as well.
>
> Sorry, I didn't actually have an analysis for module BTF, I was just
> extrapolating the result I had seen for vmlinux. I went ahead and did a
> proper test, generating BTF for a distribution kernel from Oracle Linux
> (kernel-uek-5.15.0-1.43.4.1.el9uek.x86_64) - something that I easily had
> on hand and could regenerate the BTF for quickly.
>
> Basically, the steps were:
>
>     pahole -J vmlinux --btf_encode_detached=vmlinux.btf
>     pahole -J vmlinux --btf_encode_detached=vmlinux.btf.all \
>            --encode_all_btf_vars
>
>     # For each module
>     pahole -J $MODULE --btf_encode_detached=$MODULE.btf \
>            --btf_base=vmlinux.btf
>     pahole -J $MODULE --btf_encode_detached=$MODULE.btf.all \
>            --btf_base=vmlinux.btf --encode_all_btf_vars
>
>     # what if we based the module BTF on the "vmlinux.btf.all" instead?
>     pahole -J $MODULE --btf_encode_detached=$MODULE.btf.all.all \
>            --btf_base=vmlinux.btf.all --encode_all_btf_vars
>
> And then using ls/awk to sum up the bytes of each BTF file. Results are:
>
> vmlinux:
>
> -rw-r-----. 1 opc opc 4904193 Sep  9 18:58 vmlinux.btf
> -rw-r-----. 1 opc opc 6534684 Sep  9 18:58 vmlinux.btf.all
>
> In this case there's a 33% increase in BTF size.
>
> modules:
>
> $ ls -l *.btf | awk '{sum += $5} END {print(sum)}'
> 43979532
> $ ls -l *.btf.all | awk '{sum += $5} END {print(sum)}'
> 44757792
> $ ls -l *.btf.all.all | awk '{sum += $5} END {print(sum)}'
> 44696639
>
> So the "*.btf.all.all" modules were just an experiment to see if the
> extra data inside "vmlinux.btf.all" could reduce some duplication in
> module BTF. The answer was yes, but not enough to make up for the
> increase in the vmlinux BTF size.
>
> The "*.btf.all" modules are the ones we would actually expect to use in
> Option #1, where we have a vmlinux-btf-extras and the rest of the
> modules include their globals in their BTF sections directly, and are
> based off of the vmlinux BTF. This test shows on average, that the
> module BTF size would grow by 1.6% with Option #1. Of course the exact
> memory size that accounts for will vary by workload, depending on how
> many modules are loaded. But I'd imagine, assuming you have around 5MB
> of module BTF *actually loaded*, then the overhead would be around 85k
> bytes.  I don't know about how you feel, but I think that sounds
> acceptable, it's just 22 pages at 4k size :)
>
> Let me know how it sounds to you.
>
> Thanks,
> Stephen
>
> >>
> >>
> >> Option #2
> >> ---------
> >>
> >> * The vmlinux-btf-extra module is still added as in Option #1.
> >>
> >> * Further, each module would have its own "$MODULE-btf-extra" module to
> >>   add in extra BTF. These would be built with a --btf_base=$MODULE.ko
> >>   and of course that BTF is based on vmlinux, so we would have:
> >>
> >>   vmlinux_btf              [ functions and percpu vars only ]
> >>   |- vmlinux-btf-extras    [ all other vars for vmlinux ]
> >>   |- $MODULE               [ functions and percpu vars only ]
> >>      |- $MODULE-btf-extra  [ all  other vars for $MODULE ]
> >>
> >> This is much more complex, pahole must be extended to support a
> >> hierarchy of --btf_base files. The kernel itself may not need to
> >> understand multi-level BTF since there's no requirement that it actually
> >> understand $MODULE-btf-extra, so long as it exposes it via
> >> /sys/kernel/btf/$MODULE-btf-extra. I'd also like to see some sort of
> >> mechanism to allow an administrator to say "please always load
> >> $MODULE-btf-extras alongside $MODULE", but I think that would be a
> >> userspace problem.
> >>
> >> This resolves issue (a) from option #1, of course at implementation
> >> cost.
> >>
> >> Regardless of Option #1 or #2, I'd propose that we implement this as a
> >> tristate, similar to what Alan proposed [2]. When set to "m" we use the
> >> solutions described above, and when set to "y", we don't bother with it,
> >> instead using --encode_all_btf_vars for all generation.
> >>
> >> If we go with Option #1, no changes to this series should be necessary.
> >> If we go with Option #2, I'll need to extend pahole to support at least
> >> two BTF base files. Please let me know your thoughts.
> >
> > Completely agree that two level btf-extra needs quite a bit more work.
> > Before we proceed with option 2 let's figure out
> > the reason for extra space in option 1.

I don't think an extra module for each module just for keeping those
all-var-BTFs is acceptable, so Option #2 doesn't even seem like an
option.

But given a very small increase in size of BTF for modules when
including variables, I think Option #1 is quite reasonable.



[Index of Archives]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux