On Fri, Aug 7, 2020 at 11:41 AM Andrii Nakryiko <andrii.nakryiko@xxxxxxxxx> wrote: > > On Fri, Aug 7, 2020 at 10:24 AM Jean-Philippe Brucker > <jean-philippe@xxxxxxxxxx> wrote: > > > > Hi, > > > > [Adding the linux-arm-kernel list on Cc] > > > > On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote: > > > Hi everyone, > > > > > > recently we have begun extensive research into eBPF and related > > > technologies. Seeking an easier development process, we have switched over > > > to using the eBPF CO-RE [0] approach internally which has enabled us to > > > simplify most aspects of eBPF development, especially those related to > > > cross-compilation. > > > > > > However, as part of these efforts we have stumbled upon several problems > > > that we feel would benefit from a community discussion where we may share > > > our solutions and discuss alternatives moving forward. > > > > > > As a reference point, we have started researching and modifying several eBPF > > > CO-RE samples that have been developed or migrated from existing `bcc` > > > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools` > > > directory [1]. Some of these samples have just recently been converted to > > > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has > > > proven to be very interesting. > > > > > > First showstopper for cross-compiling aforementioned example on the ARM > > > 32-bit platform has been with regards to generation of the required > > > `vmlinux.h` kernel header from the BTF information. More specifically, our > > > initial approach to have e.g. a compilation target dependency which would > > > invoke `bpftool` at configure time was not appropriate due to several > > > issues: a) CO-RE requires host kernel to have been compiled in such a way to > > > expose BTF information which may not available, and b) the generated > > > `vmlinux.h` was actually architecture-specific. > > > > > > The second point proved interesting because `tcpconnect` makes use of the > > > `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as > > > the first function parameter. The `pt_regs` structure is defined by the > > > kernel and is architecture-specific. Since `libbpf` does have > > > architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h` > > > resulted in cross-compilation failure as `libbpf` provided macros that work > > > with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To > > > resolve this issue, we have resorted to including pre-generated > > > `<arch>_vmlinux.h` files in our CO-RE build system. > > > > > > However, there are certainly drawbacks to this approach: a) (relatively) > > > large file size of the generated headers, b) regular maintenance to > > > re-generate the header files for various architectures and kernel versions, > > > and c) incompatible definitions being generated, to name a few. This last > > > point relates to the the fact that our `aarch64`/`arm64` kernel generates > > > the following definition using `bpftool`, which has resulted in compilation > > > failure: > > > > > > ``` > > > typedef __Poly8_t poly8x16_t[16]; > > > ``` > > > > > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We > > > have opted to comment out this line as there was no additional `poly8x16_t` > > > usage in the header file. > > > > It looks like this "__Poly8_t" type is internal to GCC (provided in > > arm_neon.h) and clang has its own internals. I managed to reproduce this > > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at > > the moment. Maybe libbpf should generate defines to translate these > > intrinsics between clang and gcc? Not very elegant. I'll take another > > look next week. > > libbpf is already blacklisting __builtin_va_list for GCC, so we can > just add __Poly8_t to the list. See [0]. > Are there any other types like that? If you guys can provide me this, > I'll gladly update libbpf to take those compiler-provided > types/built-ins into account. Shouldn't __Int8x16_t and friends cause the same trouble? There is a bunch more in gcc/config/arm/arm-simd-builtin-types.def. May be there is a way to detect compiler builtin types by pattern matching their dwarf/btf shape and skip them automatically? The simplest, of course, is to only add a few that caused this known trouble to blocklist.
