On Wed, Feb 26, 2025 at 6:20 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: > > On 25/02/2025 21:52, Andrii Nakryiko wrote: > > On Tue, Feb 25, 2025 at 2:02 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: > >> > >> On 07/02/2025 23:50, Alexei Starovoitov wrote: > >>> On Thu, Feb 6, 2025 at 5:21 PM Stephen Brennan > >>> <stephen.s.brennan@xxxxxxxxxx> wrote: > >>>> When the feature was implemented in pahole, my measurements indicated > >>>> that vmlinux BTF size increased by about 25.8%, and module BTF size > >>>> increased by 53.2%. Due to these increases, the feature is implemented > >>>> behind a new config option, allowing users sensitive to increased memory > >>>> usage to disable it. > >>>> > >>> > >>> ... > >>>> +config DEBUG_INFO_BTF_GLOBAL_VARS > >>>> + bool "Generate BTF type information for all global variables" > >>>> + default y > >>>> + depends on DEBUG_INFO_BTF && PAHOLE_VERSION >= 128 > >>>> + help > >>>> + Include type information for all global variables in the BTF. This > >>>> + increases the size of the BTF information, which increases memory > >>>> + usage at runtime. With global variable types available, runtime > >>>> + debugging and tracers may be able to provide more detail. > >>> > >>> This is not a solution. > >>> Even if it's changed to 'default n' distros will enable it > >>> like they enable everything and will suffer a regression. > >>> > >>> We need to add a new module like vmlinux_btf.ko that will contain > >>> this additional BTF data. For global vars and everything else we might need. > >>> > >> > >> In this area, I've been exploring adding support for > >> CONFIG_DEBUG_INFO_BTF=m , so that the BTF info for vmlinux is delivered > >> via a module. From the consumer side, everything looks identical > >> (/sys/kernel/btf/vmlinux is there etc), it is just that the .BTF section > >> is delivered via btf_vmlinux.ko instead. The original need for this was > >> that embedded folks noted that because in the current situation BTF data > >> is in vmlinux, they cannot enable BTF because such small-footprint > >> systems do not support a large vmlinux binary. However they could > >> potentially use kernel BTF if it was delivered via a module. The other > >> nice thing about module delivery in the general case is we can make use > >> of module compression. In experiments I see a 5.8Mb vmlinux BTF reduce > >> to a 1.8Mb btf_vmlinux.ko.gz module on-disk. > >> > >> The challenge in delivering vmlinux BTF in a module is that on module > >> load during boot other modules expect vmlinux BTF to be there when > >> adding their own BTF to /sys/kernel/btf. And kfunc registration from > >> kernel and modules expects this also. So support for deferred BTF module > >> load/kfunc registration is required too. I've implemented the former and > >> now am working on the latter. Hope to have some RFC patches ready soon, > >> but it looks feasible at this point. > > > > Lazy btf_vmlinux.ko loading when BTF is actually needed (i.e., when > > user reads /sys/kernel/btf/vmlinux for the first time; or when BPF > > program is validated and needs kernel BTF) would be great. Curious too > > see how all that fits together! > > > >> > >> Assuming such an option was available to small-footprint systems, should > >> we consider adding global variables to core vmlinux BTF along with > >> per-cpu variables? Then vmlinux BTF extras could be used for some of the > >> additional optional representations like function site-specific data > >> (inlines etc)? Or are there other factors other than on-disk footprint > >> that we need to consider? Thanks! > > > > I'd keep BTF for variables separate from "core" vmlinux BTF. We can > > have /sys/kernel/btf/vmlinux.vars, which would depend on > > /sys/kernel/btf/vmlinux as a base BTF. Separately, we could eventually > > have /sys/kernel/btf/vmlinux.inlines which would also have > > /sys/kernel/btf/vmlinux as base BTF. If no one needs vmlinux.vars on > > the system, we won't need to waste memory on it. Seems more modular > > and extensible. > > > > Sounds good. So thinking about how this fits with > CONFIG_DEBUG_INFO_BTF=m, perhaps the approach would be to use > btf_vmlinux.ko for all such extensible /sys/kernel/btf/vmlinux.vars, > vmlinux.inlines etc. Each of these is derived from .BTF.vars , > .BTF.inlines sections in btf_vmlinux.ko. These are optionally included > via CONFIG_DEBUG_INFO_BTF_EXTRAS list. If CONFIG_DEBUG_INFO_BTF=y the > core vmlinux section stays in vmlinux itself and the extras are > delivered via btf_vmlinux.ko, but if CONFIG_DEBUG_INFO_BTF=m, the > vmlinux .BTF section is delivered in btf_vmlinux.ko too. > > If this makes sense, I'll try and put together the > CONFIG_DEBUG_INFO_BTF=m support first, and that will give us a > btf_vmlinux.ko to work with for delivery of extras. Thanks! I'd keep our options open as to whether btf_vmlinux.ko contains all vmlinux BTFs (core BTF, inlines, variables) or we have a separate module for some subsets. E.g., variables, while a useful thing, probably won't be used all that frequently (i.e., only while debugging with drgn), so co-locating it with vmlinux BTF itself might be a waste in most cases. But other than that makes sense. > > Alan > > >> > >> Alan > >> > >>> pw-bot: cr > >>> > >> >
