Hi everyone,
recently we have begun extensive research into eBPF and related
technologies. Seeking an easier development process, we have switched
over to using the eBPF CO-RE [0] approach internally which has enabled
us to simplify most aspects of eBPF development, especially those
related to cross-compilation.
However, as part of these efforts we have stumbled upon several problems
that we feel would benefit from a community discussion where we may
share our solutions and discuss alternatives moving forward.
As a reference point, we have started researching and modifying several
eBPF CO-RE samples that have been developed or migrated from existing
`bcc` tooling. Most notable examples are those present in `bcc`'s
`libbpf-tools` directory [1]. Some of these samples have just recently
been converted to respective eBPF CO-RE variants, of which the
`tcpconnect` tracing sample has proven to be very interesting.
First showstopper for cross-compiling aforementioned example on the ARM
32-bit platform has been with regards to generation of the required
`vmlinux.h` kernel header from the BTF information. More specifically,
our initial approach to have e.g. a compilation target dependency which
would invoke `bpftool` at configure time was not appropriate due to
several issues: a) CO-RE requires host kernel to have been compiled in
such a way to expose BTF information which may not available, and b) the
generated `vmlinux.h` was actually architecture-specific.
The second point proved interesting because `tcpconnect` makes use of
the `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs
*ctx` as the first function parameter. The `pt_regs` structure is
defined by the kernel and is architecture-specific. Since `libbpf` does
have architecture-specific conditionals, pairing it with an "invalid"
`vmlinux.h` resulted in cross-compilation failure as `libbpf` provided
macros that work with ARM `pt_regs`, and `vmlinux.h` had an x86
`pt_regs` definition. To resolve this issue, we have resorted to
including pre-generated `<arch>_vmlinux.h` files in our CO-RE build system.
However, there are certainly drawbacks to this approach: a) (relatively)
large file size of the generated headers, b) regular maintenance to
re-generate the header files for various architectures and kernel
versions, and c) incompatible definitions being generated, to name a
few. This last point relates to the the fact that our `aarch64`/`arm64`
kernel generates the following definition using `bpftool`, which has
resulted in compilation failure:
```
typedef __Poly8_t poly8x16_t[16];
```
AFAICT these are ARM NEON intrinsic definitions which are GCC-specific.
We have opted to comment out this line as there was no additional
`poly8x16_t` usage in the header file.
Given various issues we have encountered so far (among which is a kernel
panic/crash on a specific device), additional input and feedback
regarding cross-compilation of the eBPF utilities would be greatly
appreciated.
[0]
https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
[1] https://github.com/iovisor/bcc/tree/master/libbpf-tools
Best regards,
Sartura eBPF Team
