Thanks for the quick response. > > Greeting, > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and libbpf_0.8.1. > > Please kindly let me know if this is not the right group to ask, since I'm new. > > > > To give context of this question: > > This system has limited disk size, doesn't need the CO-RE feature, > > and has all debug symbols stripped in release build. Having an extra > > btf/vmlinux file might be problematic, disk-wise. > Thanks for getting in touch - ideally I think we'd like to be > able to support BTF even on small systems. It would probably > help to understand what space constraints you have - is it just > disk space, or are disk space and memory highly constrained? The > mechanics of BTF are that it is generated and then embedded in the vmlinux > binary in a .BTF section. The BTF info is then exposed at runtime > via a /sys/kernel/btf/vmlinux pseudo-file. So when assessing overhead, > there are two questions to ask I think: > 1. how does BTF inclusion effect disk space? > 2. how does BTF inclusion effect memory footprint? > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb. > However, an important thing to bear in mind is that it is in the > vmlinux binary, that on most space-constrained systems is compressed > to /boot/vmlinuz-<VERSION>. When I compress the BTF by hand, it reduces > by a factor of around 6, so a ballpark figure is around 1.5Mb of > the vmlinuz binary on-disk, which equates to around 10% of the overall > binary size in my case. Your results may vary, especially if > a lot of CONFIG options are switched off (as they might be on a > space-sensitive system). > For memory footprint, BTF will be extracted from the .BTF section > and will then take up around 6Mb. > Another piece of the puzzle is module BTF - it contains the > per-module type info not in the core kernel, but again if modules > are compressed, on-disk storage might not be a massive issue. > Anyway, hopefully the above gives you a sense for the kinds of > costs BTF has. Thank you. This information on disk and memory is really helpful. At this moment, I'm only looking at disk-size. > > > > Question 1> > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later), without > > CONFIG_DEBUG_INFO_BTF ? > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have > > /sys/kernel/btf/vmlinux removed. > > > It really depends on what you're planning on doing. > BTF has become central to a lot of aspects of BPF; higher-performance > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF > soon I believe. > So if you're using BPF without BTF, there are generally ways to make things work (using kprobes instead of fentry for example), but you > will have less options. I seem to recall some fixes landed to > ensure that absence of BTF shouldn't prevent program loading in > cases where BTF is not needed. If you run into any such failures, > I'd suggest reporting them and hopefully we can get them fixed. I have a follow up question on how CO-RE uses BTF: where exactly does the relocation happen ? It seems, in theory, it can happens in two places: 1> from libBPF at user space 2> from kernel https://nakryiko.com/posts/bpf-portability-and-co-re/ " It takes compiled BPF ELF object file, post-processes it as necessary, sets up various kernel objects (maps, programs, etc), and triggers BPF program loading and verification." I assume there is a syscall to provide BTF information from kernel to user space, and libBPF uses that info to post-processing the ELF file. Is there a sample BPF code with explanation of a sequence of actions done by libBPF (roughly) to look at ? And why do maps need to be relocated ? 2> https://nakryiko.com/posts/bpf-core-reference-guide/ BTF-enabled BPF program types with direct memory reads In this mode, is that kernel doing relocation ? or is that still libBPF? For example: how/where vma->vm_start is relocated. SEC("lsm/file_mprotect") int BPF_PROG(mprotect_audit, struct vm_area_struct *vma, unsigned long reqprot, unsigned long prot, int ret) { /* .. omit ..*/ int is_heap; is_heap = (vma->vm_start >= vma->vm_mm->start_brk && vma->vm_end <= vma->vm_mm->brk); /* .. omit .. */ } Thanks Best Regards, Jeff Xu On Fri, Sep 9, 2022 at 8:29 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: > > On 09/09/2022 06:22, Jeff Xu wrote: > > Greeting, > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and libbpf_0.8.1. > > Please kindly let me know if this is not the right group to ask, since I'm new. > > > > To give context of this question: > > This system has limited disk size, doesn't need the CO-RE feature, > > and has all debug symbols stripped in release build. Having an extra > > btf/vmlinux file might be problematic, disk-wise. > > Thanks for getting in touch - ideally I think we'd like to be > able to support BTF even on small systems. It would probably > help to understand what space constraints you have - is it just > disk space, or are disk space and memory highly constrained? The > mechanics of BTF are that it is generated and then embedded in the vmlinux > binary in a .BTF section. The BTF info is then exposed at runtime > via a /sys/kernel/btf/vmlinux pseudo-file. So when assessing overhead, > there are two questions to ask I think: > > 1. how does BTF inclusion effect disk space? > 2. how does BTF inclusion effect memory footprint? > > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb. > However, an important thing to bear in mind is that it is in the > vmlinux binary, that on most space-constrained systems is compressed > to /boot/vmlinuz-<VERSION>. When I compress the BTF by hand, it reduces > by a factor of around 6, so a ballpark figure is around 1.5Mb of > the vmlinuz binary on-disk, which equates to around 10% of the overall > binary size in my case. Your results may vary, especially if > a lot of CONFIG options are switched off (as they might be on a > space-sensitive system). > > For memory footprint, BTF will be extracted from the .BTF section > and will then take up around 6Mb. > > Another piece of the puzzle is module BTF - it contains the > per-module type info not in the core kernel, but again if modules > are compressed, on-disk storage might not be a massive issue. > > Anyway, hopefully the above gives you a sense for the kinds of > costs BTF has. > > > > > Question 1> > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later), without > > CONFIG_DEBUG_INFO_BTF ? > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have > > /sys/kernel/btf/vmlinux removed. > > > > It really depends what you're planning on doing. > > BTF has become central to a lot of aspects of BPF; higher-performance > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF > soon I believe. > > So if you're using BPF without BTF, there are generally ways to make > things work (using kprobes instead of fentry for example), but you > will have less options. I seem to recall some fixes landed to > ensure that absence of BTF shouldn't prevent program loading in > cases where BTF is not needed. If you run into any such failures, > I'd suggest reporting them and hopefully we can get them fixed. > > > Question 2: From debug information included at run time point of view, > > (1) having btf/vmlinux vs (2) kernel build with > > CONFIG_DEBUG_INFO_DWARF5 but not stripped, > > are those two contains the same amount of debug information at runtime? > > > > DWARF5 will contain more debug info, but will likely have a larger footprint > as a consequence. I'd suggest running the experiment yourself to compare. > > > Question 3: Will libbpf + btf/vmlinx, break expectation of kernel ASLR > > feature ? I assume it shouldn't, but would like to double check. > > > > Nope, no issue here that I'm aware of. I've used KASLR + BTF and haven't seen > any problems at least. > > > Thanks > > Best Regards, > > Jeff Xu > >
