On Mon, Sep 12, 2022 at 8:41 PM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote: > > On Mon, Sep 12, 2022 at 5:17 PM Jeff Xu <jeffxu@xxxxxxxxxxxx> wrote: > > > > On Sun, Sep 11, 2022 at 4:36 PM Jeff Xu <jeffxu@xxxxxxxxxxxx> wrote: > > > > > > Thanks for the quick response. > > > > > > > > Greeting, > > > > > > > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and libbpf_0.8.1. > > > > > Please kindly let me know if this is not the right group to ask, since I'm new. > > > > > > > > > > To give context of this question: > > > > > This system has limited disk size, doesn't need the CO-RE feature, > > > > > and has all debug symbols stripped in release build. Having an extra > > > > > btf/vmlinux file might be problematic, disk-wise. > > > > > > > Thanks for getting in touch - ideally I think we'd like to be > > > > able to support BTF even on small systems. It would probably > > > > help to understand what space constraints you have - is it just > > > > disk space, or are disk space and memory highly constrained? The > > > > mechanics of BTF are that it is generated and then embedded in the vmlinux > > > > binary in a .BTF section. The BTF info is then exposed at runtime > > > > via a /sys/kernel/btf/vmlinux pseudo-file. So when assessing overhead, > > > > there are two questions to ask I think: > > > > > > > 1. how does BTF inclusion effect disk space? > > > > 2. how does BTF inclusion effect memory footprint? > > > > > > > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb. > > > > However, an important thing to bear in mind is that it is in the > > > > vmlinux binary, that on most space-constrained systems is compressed > > > > to /boot/vmlinuz-<VERSION>. When I compress the BTF by hand, it reduces > > > > by a factor of around 6, so a ballpark figure is around 1.5Mb of > > > > the vmlinuz binary on-disk, which equates to around 10% of the overall > > > > binary size in my case. Your results may vary, especially if > > > > a lot of CONFIG options are switched off (as they might be on a > > > > space-sensitive system). > > > > > > > For memory footprint, BTF will be extracted from the .BTF section > > > > and will then take up around 6Mb. > > > > > > > Another piece of the puzzle is module BTF - it contains the > > > > per-module type info not in the core kernel, but again if modules > > > > are compressed, on-disk storage might not be a massive issue. > > > > > > > Anyway, hopefully the above gives you a sense for the kinds of > > > > costs BTF has. > > > > > > Thank you. This information on disk and memory is really helpful. > > > At this moment, I'm only looking at disk-size. > > > > > > > > > > > > > Question 1> > > > > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later), without > > > > > CONFIG_DEBUG_INFO_BTF ? > > > > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have > > > > > /sys/kernel/btf/vmlinux removed. > > > > > > > > > > > > It really depends on what you're planning on doing. > > > > > > > BTF has become central to a lot of aspects of BPF; higher-performance > > > > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF > > > > soon I believe. > > > > > > > So if you're using BPF without BTF, there are generally ways to make > > > things work (using kprobes instead of fentry for example), but you > > > > will have less options. I seem to recall some fixes landed to > > > > ensure that absence of BTF shouldn't prevent program loading in > > > > cases where BTF is not needed. If you run into any such failures, > > > > I'd suggest reporting them and hopefully we can get them fixed. > > > > > > I have a follow up question on how CO-RE uses BTF: where exactly does > > > the relocation happen ? > > > It seems, in theory, it can happens in two places: 1> from libBPF at > > > user space 2> from kernel > > > > > > https://nakryiko.com/posts/bpf-portability-and-co-re/ > > > " It takes compiled BPF ELF object file, post-processes it as > > > necessary, sets up various kernel objects (maps, programs, etc), > > > and triggers BPF program loading and verification." > > > > > > I assume there is a syscall to provide BTF information from kernel to > > > user space, and libBPF uses that info to post-processing the ELF file. > > > > > > Is there a sample BPF code with explanation of a sequence of actions > > > done by libBPF (roughly) to look at ? > > > And why do maps need to be relocated ? > > > > > > 2> > > > https://nakryiko.com/posts/bpf-core-reference-guide/ BTF-enabled BPF > > > program types with direct memory reads > > > In this mode, is that kernel doing relocation ? or is that still libBPF? > > > For example: how/where vma->vm_start is relocated. > > > > > > SEC("lsm/file_mprotect") > > > int BPF_PROG(mprotect_audit, struct vm_area_struct *vma, > > > unsigned long reqprot, unsigned long prot, int ret) > > > { > > > /* .. omit ..*/ > > > int is_heap; > > > is_heap = (vma->vm_start >= vma->vm_mm->start_brk && > > > vma->vm_end <= vma->vm_mm->brk); > > > /* .. omit .. */ > > > } > > > > > > Thanks > > > Best Regards, > > > Jeff Xu > > > > > > > > > On Fri, Sep 9, 2022 at 8:29 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: > > > > > > > > On 09/09/2022 06:22, Jeff Xu wrote: > > > > > Greeting, > > > > > > > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and libbpf_0.8.1. > > > > > Please kindly let me know if this is not the right group to ask, since I'm new. > > > > > > > > > > To give context of this question: > > > > > This system has limited disk size, doesn't need the CO-RE feature, > > > > > and has all debug symbols stripped in release build. Having an extra > > > > > btf/vmlinux file might be problematic, disk-wise. > > > > > > > > Thanks for getting in touch - ideally I think we'd like to be > > > > able to support BTF even on small systems. It would probably > > > > help to understand what space constraints you have - is it just > > > > disk space, or are disk space and memory highly constrained? The > > > > mechanics of BTF are that it is generated and then embedded in the vmlinux > > > > binary in a .BTF section. The BTF info is then exposed at runtime > > > > via a /sys/kernel/btf/vmlinux pseudo-file. So when assessing overhead, > > > > there are two questions to ask I think: > > > > > > > > 1. how does BTF inclusion effect disk space? > > > > 2. how does BTF inclusion effect memory footprint? > > > > > > > > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb. > > > > However, an important thing to bear in mind is that it is in the > > > > vmlinux binary, that on most space-constrained systems is compressed > > > > to /boot/vmlinuz-<VERSION>. When I compress the BTF by hand, it reduces > > > > by a factor of around 6, so a ballpark figure is around 1.5Mb of > > > > the vmlinuz binary on-disk, which equates to around 10% of the overall > > > > binary size in my case. Your results may vary, especially if > > > > a lot of CONFIG options are switched off (as they might be on a > > > > space-sensitive system). > > > > > > > > For memory footprint, BTF will be extracted from the .BTF section > > > > and will then take up around 6Mb. > > > > > > > > Another piece of the puzzle is module BTF - it contains the > > > > per-module type info not in the core kernel, but again if modules > > > > are compressed, on-disk storage might not be a massive issue. > > > > > > > > Anyway, hopefully the above gives you a sense for the kinds of > > > > costs BTF has. > > > > > > > > > > > > > > Question 1> > > > > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later), without > > > > > CONFIG_DEBUG_INFO_BTF ? > > > > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have > > > > > /sys/kernel/btf/vmlinux removed. > > > > > > > > > > > > > It really depends what you're planning on doing. > > > > > > > > BTF has become central to a lot of aspects of BPF; higher-performance > > > > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF > > > > soon I believe. > > > > > > > > So if you're using BPF without BTF, there are generally ways to make > > > > things work (using kprobes instead of fentry for example), but you > > > > will have less options. I seem to recall some fixes landed to > > > > ensure that absence of BTF shouldn't prevent program loading in > > > > cases where BTF is not needed. If you run into any such failures, > > > > I'd suggest reporting them and hopefully we can get them fixed. > > > > > > > > > Question 2: From debug information included at run time point of view, > > > > > (1) having btf/vmlinux vs (2) kernel build with > > > > > CONFIG_DEBUG_INFO_DWARF5 but not stripped, > > > > > are those two contains the same amount of debug information at runtime? > > > > > > > > > > > > > DWARF5 will contain more debug info, but will likely have a larger footprint > > > > as a consequence. I'd suggest running the experiment yourself to compare. > > > > > > > > > Question 3: Will libbpf + btf/vmlinx, break expectation of kernel ASLR > > > > > feature ? I assume it shouldn't, but would like to double check. > > > > > > > > > > > > > Nope, no issue here that I'm aware of. I've used KASLR + BTF and haven't seen > > > > any problems at least. > > > > > > > > > Thanks > > > > > Best Regards, > > > > > Jeff Xu > > > > > > > > > Can I understand the BTF usage in this way ? > > > > When BTF is available in the kernel runtime, it helps in two ways: > > 1> By BTF verifier (kernel) to find the offset of a member in struct > > (no libbpf modification of BYTE code needed) > > The example usage is BTF RAW tracepoint, BFP_LSM. > > Typically, those BPF programs will includes "vmlinux.h" , and uses C > > pointer style(vma->vm_start) > > > > 2> By libbpf (user space) to post-processing BPF bytecode. > > Typically, those BPF programs doesn't need to include "vmlinux.h", and > > uses bpf_core_read, such as: > > BPF_CORE_READ(vma,vm_start) > > > > Much appreciated to confirm this is right/wrong. > > > > Does not answer your question directly :) from my limited > understanding, could be incorrect, BTF is processed at compile time > and load time, load time is processed by libbpf > So even for BTF RAW tracepoint, the relocation is happening at libbpf ? According to this post: https://mozillazg.com/2022/06/ebpf-libbpf-btf-powered-enabled-raw-tracepoint-common-questions-en.html#hidthe-difference-between-btf-raw-tracepoint-and-raw-tracepoint // btf enabled struct task_struct *task = (struct task_struct *) bpf_get_current_task_btf(); u32 ppid = task->real_parent->tgid; "The btf version can access kernel memory directly from within the ebpf program. There is no need to use a helper function like bpf_core_read or bpf_probe_read_kernel to access the kernel memory as in regular raw tracepoint:" It talks about accessing kernel memory directly, so I was reading it as the kernel is doing the relocation. > > Best Regards > > Jeff Xu
