On Sun, Sep 11, 2022 at 4:36 PM Jeff Xu <jeffxu@xxxxxxxxxxxx> wrote: > > Thanks for the quick response. > > > > Greeting, > > > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and libbpf_0.8.1. > > > Please kindly let me know if this is not the right group to ask, since I'm new. > > > > > > To give context of this question: > > > This system has limited disk size, doesn't need the CO-RE feature, > > > and has all debug symbols stripped in release build. Having an extra > > > btf/vmlinux file might be problematic, disk-wise. > > > Thanks for getting in touch - ideally I think we'd like to be > > able to support BTF even on small systems. It would probably > > help to understand what space constraints you have - is it just > > disk space, or are disk space and memory highly constrained? The > > mechanics of BTF are that it is generated and then embedded in the vmlinux > > binary in a .BTF section. The BTF info is then exposed at runtime > > via a /sys/kernel/btf/vmlinux pseudo-file. So when assessing overhead, > > there are two questions to ask I think: > > > 1. how does BTF inclusion effect disk space? > > 2. how does BTF inclusion effect memory footprint? > > > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb. > > However, an important thing to bear in mind is that it is in the > > vmlinux binary, that on most space-constrained systems is compressed > > to /boot/vmlinuz-<VERSION>. When I compress the BTF by hand, it reduces > > by a factor of around 6, so a ballpark figure is around 1.5Mb of > > the vmlinuz binary on-disk, which equates to around 10% of the overall > > binary size in my case. Your results may vary, especially if > > a lot of CONFIG options are switched off (as they might be on a > > space-sensitive system). > > > For memory footprint, BTF will be extracted from the .BTF section > > and will then take up around 6Mb. > > > Another piece of the puzzle is module BTF - it contains the > > per-module type info not in the core kernel, but again if modules > > are compressed, on-disk storage might not be a massive issue. > > > Anyway, hopefully the above gives you a sense for the kinds of > > costs BTF has. > > Thank you. This information on disk and memory is really helpful. > At this moment, I'm only looking at disk-size. > > > > > > > Question 1> > > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later), without > > > CONFIG_DEBUG_INFO_BTF ? > > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have > > > /sys/kernel/btf/vmlinux removed. > > > > > > It really depends on what you're planning on doing. > > > BTF has become central to a lot of aspects of BPF; higher-performance > > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF > > soon I believe. > > > So if you're using BPF without BTF, there are generally ways to make > things work (using kprobes instead of fentry for example), but you > > will have less options. I seem to recall some fixes landed to > > ensure that absence of BTF shouldn't prevent program loading in > > cases where BTF is not needed. If you run into any such failures, > > I'd suggest reporting them and hopefully we can get them fixed. > > I have a follow up question on how CO-RE uses BTF: where exactly does > the relocation happen ? > It seems, in theory, it can happens in two places: 1> from libBPF at > user space 2> from kernel > > https://nakryiko.com/posts/bpf-portability-and-co-re/ > " It takes compiled BPF ELF object file, post-processes it as > necessary, sets up various kernel objects (maps, programs, etc), > and triggers BPF program loading and verification." > > I assume there is a syscall to provide BTF information from kernel to > user space, and libBPF uses that info to post-processing the ELF file. > > Is there a sample BPF code with explanation of a sequence of actions > done by libBPF (roughly) to look at ? > And why do maps need to be relocated ? > > 2> > https://nakryiko.com/posts/bpf-core-reference-guide/ BTF-enabled BPF > program types with direct memory reads > In this mode, is that kernel doing relocation ? or is that still libBPF? > For example: how/where vma->vm_start is relocated. > > SEC("lsm/file_mprotect") > int BPF_PROG(mprotect_audit, struct vm_area_struct *vma, > unsigned long reqprot, unsigned long prot, int ret) > { > /* .. omit ..*/ > int is_heap; > is_heap = (vma->vm_start >= vma->vm_mm->start_brk && > vma->vm_end <= vma->vm_mm->brk); > /* .. omit .. */ > } > > Thanks > Best Regards, > Jeff Xu > > > On Fri, Sep 9, 2022 at 8:29 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: > > > > On 09/09/2022 06:22, Jeff Xu wrote: > > > Greeting, > > > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and libbpf_0.8.1. > > > Please kindly let me know if this is not the right group to ask, since I'm new. > > > > > > To give context of this question: > > > This system has limited disk size, doesn't need the CO-RE feature, > > > and has all debug symbols stripped in release build. Having an extra > > > btf/vmlinux file might be problematic, disk-wise. > > > > Thanks for getting in touch - ideally I think we'd like to be > > able to support BTF even on small systems. It would probably > > help to understand what space constraints you have - is it just > > disk space, or are disk space and memory highly constrained? The > > mechanics of BTF are that it is generated and then embedded in the vmlinux > > binary in a .BTF section. The BTF info is then exposed at runtime > > via a /sys/kernel/btf/vmlinux pseudo-file. So when assessing overhead, > > there are two questions to ask I think: > > > > 1. how does BTF inclusion effect disk space? > > 2. how does BTF inclusion effect memory footprint? > > > > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb. > > However, an important thing to bear in mind is that it is in the > > vmlinux binary, that on most space-constrained systems is compressed > > to /boot/vmlinuz-<VERSION>. When I compress the BTF by hand, it reduces > > by a factor of around 6, so a ballpark figure is around 1.5Mb of > > the vmlinuz binary on-disk, which equates to around 10% of the overall > > binary size in my case. Your results may vary, especially if > > a lot of CONFIG options are switched off (as they might be on a > > space-sensitive system). > > > > For memory footprint, BTF will be extracted from the .BTF section > > and will then take up around 6Mb. > > > > Another piece of the puzzle is module BTF - it contains the > > per-module type info not in the core kernel, but again if modules > > are compressed, on-disk storage might not be a massive issue. > > > > Anyway, hopefully the above gives you a sense for the kinds of > > costs BTF has. > > > > > > > > Question 1> > > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later), without > > > CONFIG_DEBUG_INFO_BTF ? > > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have > > > /sys/kernel/btf/vmlinux removed. > > > > > > > It really depends what you're planning on doing. > > > > BTF has become central to a lot of aspects of BPF; higher-performance > > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF > > soon I believe. > > > > So if you're using BPF without BTF, there are generally ways to make > > things work (using kprobes instead of fentry for example), but you > > will have less options. I seem to recall some fixes landed to > > ensure that absence of BTF shouldn't prevent program loading in > > cases where BTF is not needed. If you run into any such failures, > > I'd suggest reporting them and hopefully we can get them fixed. > > > > > Question 2: From debug information included at run time point of view, > > > (1) having btf/vmlinux vs (2) kernel build with > > > CONFIG_DEBUG_INFO_DWARF5 but not stripped, > > > are those two contains the same amount of debug information at runtime? > > > > > > > DWARF5 will contain more debug info, but will likely have a larger footprint > > as a consequence. I'd suggest running the experiment yourself to compare. > > > > > Question 3: Will libbpf + btf/vmlinx, break expectation of kernel ASLR > > > feature ? I assume it shouldn't, but would like to double check. > > > > > > > Nope, no issue here that I'm aware of. I've used KASLR + BTF and haven't seen > > any problems at least. > > > > > Thanks > > > Best Regards, > > > Jeff Xu > > > Can I understand the BTF usage in this way ? When BTF is available in the kernel runtime, it helps in two ways: 1> By BTF verifier (kernel) to find the offset of a member in struct (no libbpf modification of BYTE code needed) The example usage is BTF RAW tracepoint, BFP_LSM. Typically, those BPF programs will includes "vmlinux.h" , and uses C pointer style(vma->vm_start) 2> By libbpf (user space) to post-processing BPF bytecode. Typically, those BPF programs doesn't need to include "vmlinux.h", and uses bpf_core_read, such as: BPF_CORE_READ(vma,vm_start) Much appreciated to confirm this is right/wrong. Best Regards Jeff Xu
