Re: BTF and libBPF

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Sep 12, 2022 at 8:41 PM Vincent Li <vincent.mc.li@xxxxxxxxx> wrote:
>
> On Mon, Sep 12, 2022 at 5:17 PM Jeff Xu <jeffxu@xxxxxxxxxxxx> wrote:
> >
> > On Sun, Sep 11, 2022 at 4:36 PM Jeff Xu <jeffxu@xxxxxxxxxxxx> wrote:
> > >
> > > Thanks for the quick response.
> > >
> > > > > Greeting,
> > > > >
> > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and  libbpf_0.8.1.
> > > > > Please kindly let me know if this is not the right group to ask, since I'm new.
> > > > >
> > > > > To give context of this question:
> > > > > This system has limited disk size, doesn't need the CO-RE feature,
> > > > > and has all debug symbols stripped in release build.   Having an extra
> > > > > btf/vmlinux file might be problematic, disk-wise.
> > >
> > > > Thanks for getting in touch - ideally I think we'd like to be
> > > > able to support BTF even on small systems. It would probably
> > > > help to understand what space constraints you have - is it just
> > > > disk space, or are disk space and memory highly constrained? The
> > > > mechanics of BTF are that it is generated and then embedded in the vmlinux
> > > > binary in a .BTF section. The BTF info is then exposed at runtime
> > > > via a /sys/kernel/btf/vmlinux pseudo-file.  So when assessing overhead,
> > > > there are two questions to ask I think:
> > >
> > > > 1. how does BTF inclusion effect disk space?
> > > > 2. how does BTF inclusion effect memory footprint?
> > >
> > > > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb.
> > > > However, an important thing to bear in mind is that it is in the
> > > > vmlinux binary, that on most space-constrained systems is compressed
> > > > to /boot/vmlinuz-<VERSION>.  When I compress the BTF by hand, it reduces
> > > > by a factor of around 6, so a ballpark figure is around 1.5Mb of
> > > > the vmlinuz binary on-disk, which equates to around 10% of the overall
> > > > binary size in my case. Your results may vary, especially if
> > > > a lot of CONFIG options are switched off (as they might be on a
> > > > space-sensitive system).
> > >
> > > > For memory footprint, BTF will be extracted from the .BTF section
> > > > and will then take up around 6Mb.
> > >
> > > > Another piece of the puzzle is module BTF - it contains the
> > > > per-module type info not in the core kernel, but again if modules
> > > > are compressed, on-disk storage might not be a massive issue.
> > >
> > > > Anyway, hopefully the above gives you a sense for the kinds of
> > > > costs BTF has.
> > >
> > > Thank you. This information on disk and memory is really helpful.
> > > At this moment, I'm only looking at disk-size.
> > >
> > > > >
> > > > > Question 1>
> > > > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later),  without
> > > > > CONFIG_DEBUG_INFO_BTF ?
> > > > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have
> > > > > /sys/kernel/btf/vmlinux removed.
> > > > >
> > >
> > > > It really depends on what you're planning on doing.
> > >
> > > > BTF has become central to a lot of aspects of BPF; higher-performance
> > > > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF
> > > > soon I believe.
> > >
> > > > So if you're using BPF without BTF, there are generally ways to make
> > > things work (using kprobes instead of fentry for example), but you
> > > > will have less options.  I seem to recall some fixes landed to
> > > > ensure that absence of BTF shouldn't prevent program loading in
> > > > cases where BTF is not needed. If you run into any such failures,
> > > > I'd suggest reporting them and hopefully we can get them fixed.
> > >
> > > I have a follow up question on how CO-RE uses BTF: where exactly does
> > > the relocation happen ?
> > > It seems, in theory,  it can happens in two places: 1> from libBPF at
> > > user space 2> from kernel
> > >
> > > https://nakryiko.com/posts/bpf-portability-and-co-re/
> > > " It takes compiled BPF ELF object file, post-processes it as
> > > necessary, sets up various kernel objects (maps, programs, etc),
> > > and triggers BPF program loading and verification."
> > >
> > > I assume there is a syscall to provide BTF information from kernel to
> > > user space, and libBPF uses that info to post-processing the ELF file.
> > >
> > > Is there a sample BPF code with explanation of a sequence of actions
> > > done by libBPF (roughly) to look at ?
> > > And why do maps need to be relocated ?
> > >
> > > 2>
> > > https://nakryiko.com/posts/bpf-core-reference-guide/ BTF-enabled BPF
> > > program types with direct memory reads
> > > In this mode, is that kernel doing relocation ? or is that still libBPF?
> > > For example: how/where vma->vm_start is relocated.
> > >
> > > SEC("lsm/file_mprotect")
> > > int BPF_PROG(mprotect_audit, struct vm_area_struct *vma,
> > >     unsigned long reqprot, unsigned long prot, int ret)
> > > {
> > >    /* .. omit ..*/
> > > int is_heap;
> > > is_heap = (vma->vm_start >= vma->vm_mm->start_brk &&
> > >   vma->vm_end <= vma->vm_mm->brk);
> > >    /* .. omit .. */
> > > }
> > >
> > > Thanks
> > > Best Regards,
> > > Jeff Xu
> > >
> > >
> > > On Fri, Sep 9, 2022 at 8:29 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
> > > >
> > > > On 09/09/2022 06:22, Jeff Xu wrote:
> > > > > Greeting,
> > > > >
> > > > > I have questions related to CONFIG_DEBUG_INFO_BTF, and  libbpf_0.8.1.
> > > > > Please kindly let me know if this is not the right group to ask, since I'm new.
> > > > >
> > > > > To give context of this question:
> > > > > This system has limited disk size, doesn't need the CO-RE feature,
> > > > > and has all debug symbols stripped in release build.   Having an extra
> > > > > btf/vmlinux file might be problematic, disk-wise.
> > > >
> > > > Thanks for getting in touch - ideally I think we'd like to be
> > > > able to support BTF even on small systems. It would probably
> > > > help to understand what space constraints you have - is it just
> > > > disk space, or are disk space and memory highly constrained? The
> > > > mechanics of BTF are that it is generated and then embedded in the vmlinux
> > > > binary in a .BTF section. The BTF info is then exposed at runtime
> > > > via a /sys/kernel/btf/vmlinux pseudo-file.  So when assessing overhead,
> > > > there are two questions to ask I think:
> > > >
> > > > 1. how does BTF inclusion effect disk space?
> > > > 2. how does BTF inclusion effect memory footprint?
> > > >
> > > > For 1, on a recent bpf-next kernel, core vmlinux BTF is around 6Mb.
> > > > However, an important thing to bear in mind is that it is in the
> > > > vmlinux binary, that on most space-constrained systems is compressed
> > > > to /boot/vmlinuz-<VERSION>.  When I compress the BTF by hand, it reduces
> > > > by a factor of around 6, so a ballpark figure is around 1.5Mb of
> > > > the vmlinuz binary on-disk, which equates to around 10% of the overall
> > > > binary size in my case. Your results may vary, especially if
> > > > a lot of CONFIG options are switched off (as they might be on a
> > > > space-sensitive system).
> > > >
> > > > For memory footprint, BTF will be extracted from the .BTF section
> > > > and will then take up around 6Mb.
> > > >
> > > > Another piece of the puzzle is module BTF - it contains the
> > > > per-module type info not in the core kernel, but again if modules
> > > > are compressed, on-disk storage might not be a massive issue.
> > > >
> > > > Anyway, hopefully the above gives you a sense for the kinds of
> > > > costs BTF has.
> > > >
> > > > >
> > > > > Question 1>
> > > > > Will libbpf_0.8.1(or later) work with kernel 5.10 (or later),  without
> > > > > CONFIG_DEBUG_INFO_BTF ?
> > > > > Or work with kernel compiled with CONFIG_DEBUG_INFO_BTF but have
> > > > > /sys/kernel/btf/vmlinux removed.
> > > > >
> > > >
> > > > It really depends what you're planning on doing.
> > > >
> > > > BTF has become central to a lot of aspects of BPF; higher-performance
> > > > fentry/fexit() BPF programs, CO-RE, and even XDP will be using BTF
> > > > soon I believe.
> > > >
> > > > So if you're using BPF without BTF, there are generally ways to make
> > > > things work (using kprobes instead of fentry for example), but you
> > > > will have less options.  I seem to recall some fixes landed to
> > > > ensure that absence of BTF shouldn't prevent program loading in
> > > > cases where BTF is not needed. If you run into any such failures,
> > > > I'd suggest reporting them and hopefully we can get them fixed.
> > > >
> > > > >  Question 2: From debug information included at run time point of view,
> > > > > (1) having btf/vmlinux vs (2) kernel build with
> > > > > CONFIG_DEBUG_INFO_DWARF5 but not stripped,
> > > > > are those two contains the same amount of debug information at runtime?
> > > > >
> > > >
> > > > DWARF5 will contain more debug info, but will likely have a larger footprint
> > > > as a consequence. I'd suggest running the experiment yourself to compare.
> > > >
> > > > > Question 3: Will libbpf + btf/vmlinx, break expectation of kernel ASLR
> > > > > feature ? I assume it shouldn't, but would like to double check.
> > > > >
> > > >
> > > > Nope, no issue here that I'm aware of. I've used KASLR + BTF and haven't seen
> > > > any problems at least.
> > > >
> > > > > Thanks
> > > > > Best Regards,
> > > > > Jeff Xu
> > > > >
> >
> > Can I understand the BTF usage in this way ?
> >
> > When BTF is available in the kernel runtime, it helps in two ways:
> > 1> By BTF verifier (kernel) to find the offset of a member in struct
> > (no libbpf modification of BYTE code needed)
> > The example usage is BTF RAW tracepoint, BFP_LSM.
> > Typically, those BPF programs will includes "vmlinux.h" , and  uses C
> > pointer style(vma->vm_start)
> >
> > 2> By libbpf (user space) to post-processing BPF bytecode.
> > Typically, those BPF programs doesn't need to include "vmlinux.h", and
> > uses bpf_core_read, such as:
> > BPF_CORE_READ(vma,vm_start)
> >
> > Much appreciated to confirm this is right/wrong.
> >
>
> Does not answer your question directly :) from my limited
> understanding, could be incorrect, BTF is processed at compile time
> and load time,  load time is processed  by libbpf
>
So even for BTF RAW tracepoint, the relocation is happening at libbpf ?
According to this post:
https://mozillazg.com/2022/06/ebpf-libbpf-btf-powered-enabled-raw-tracepoint-common-questions-en.html#hidthe-difference-between-btf-raw-tracepoint-and-raw-tracepoint

// btf enabled
struct task_struct *task = (struct task_struct *) bpf_get_current_task_btf();
u32 ppid = task->real_parent->tgid;

"The btf version can access kernel memory directly from within the ebpf program.
There is no need to use a helper function like bpf_core_read or
bpf_probe_read_kernel to access the kernel memory as in regular raw
tracepoint:"

It talks about accessing kernel memory directly, so I was reading it
as  the kernel is doing the relocation.

> > Best Regards
> > Jeff Xu



[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]


  Powered by Linux