On Fri, Aug 7, 2020 at 10:24 AM Jean-Philippe Brucker <jean-philippe@xxxxxxxxxx> wrote: > > Hi, > > [Adding the linux-arm-kernel list on Cc] > > On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote: > > Hi everyone, > > > > recently we have begun extensive research into eBPF and related > > technologies. Seeking an easier development process, we have switched over > > to using the eBPF CO-RE [0] approach internally which has enabled us to > > simplify most aspects of eBPF development, especially those related to > > cross-compilation. > > > > However, as part of these efforts we have stumbled upon several problems > > that we feel would benefit from a community discussion where we may share > > our solutions and discuss alternatives moving forward. > > > > As a reference point, we have started researching and modifying several eBPF > > CO-RE samples that have been developed or migrated from existing `bcc` > > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools` > > directory [1]. Some of these samples have just recently been converted to > > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has > > proven to be very interesting. > > > > First showstopper for cross-compiling aforementioned example on the ARM > > 32-bit platform has been with regards to generation of the required > > `vmlinux.h` kernel header from the BTF information. More specifically, our > > initial approach to have e.g. a compilation target dependency which would > > invoke `bpftool` at configure time was not appropriate due to several > > issues: a) CO-RE requires host kernel to have been compiled in such a way to > > expose BTF information which may not available, and b) the generated > > `vmlinux.h` was actually architecture-specific. > > > > The second point proved interesting because `tcpconnect` makes use of the > > `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as > > the first function parameter. The `pt_regs` structure is defined by the > > kernel and is architecture-specific. Since `libbpf` does have > > architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h` > > resulted in cross-compilation failure as `libbpf` provided macros that work > > with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To > > resolve this issue, we have resorted to including pre-generated > > `<arch>_vmlinux.h` files in our CO-RE build system. > > > > However, there are certainly drawbacks to this approach: a) (relatively) > > large file size of the generated headers, b) regular maintenance to > > re-generate the header files for various architectures and kernel versions, > > and c) incompatible definitions being generated, to name a few. This last > > point relates to the the fact that our `aarch64`/`arm64` kernel generates > > the following definition using `bpftool`, which has resulted in compilation > > failure: > > > > ``` > > typedef __Poly8_t poly8x16_t[16]; > > ``` > > > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We > > have opted to comment out this line as there was no additional `poly8x16_t` > > usage in the header file. > > It looks like this "__Poly8_t" type is internal to GCC (provided in > arm_neon.h) and clang has its own internals. I managed to reproduce this > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at > the moment. Maybe libbpf should generate defines to translate these > intrinsics between clang and gcc? Not very elegant. I'll take another > look next week. libbpf is already blacklisting __builtin_va_list for GCC, so we can just add __Poly8_t to the list. See [0]. Are there any other types like that? If you guys can provide me this, I'll gladly update libbpf to take those compiler-provided types/built-ins into account. [0] https://github.com/torvalds/linux/blob/master/tools/lib/bpf/btf_dump.c#L585-L598 > > > Given various issues we have encountered so far (among which is a kernel > > panic/crash on a specific device), additional input and feedback regarding > > cross-compilation of the eBPF utilities would be greatly appreciated. > > I don't know if there is a room for improvement regarding your a) and b) > points, as I think the added complexity is inherent to cross-building. But > kernel crashes definitely need to be fixed, as well as the above problem. > > Thanks, > Jean
