Introduce BPF trampoline that works as a bridge between kernel functions and BPF programs. The first use case is fentry/fexit BPF programs that are roughly equivalent to kprobe/kretprobe. Unlike k[ret]probe there is practically zero overhead to call a set of BPF programs before or after kernel function. In the future patches networking use cases will be explored. For example: BPF trampoline can be used to call XDP programs from drivers with direct calls or wrapping BPF program into another BPF program. The patch set depends on register_ftrace_direct() API. It's not upstream yet and available in [1]. The first patch is a hack to workaround this dependency. The idea is to land this set via bpf-next tree and land register_ftrace_direct via Steven's ftrace tree. Then during the merge window revert the first patch. Steven, do you think it's workable? As an alternative we can route register_ftrace_direct patches via bpf-next ? Peter's static_call patches [2] are solving the issue of indirect call overhead for large class of kernel use cases, but unfortunately don't help calling into a set of BPF programs. BPF trampoline's first goal is to translate kernel calling convention into BPF calling convention. The second goal is to call a set of programs efficiently. In the future we can replace BPF_PROG_RUN_ARRAY with BPF trampoline variant. Another future work is to add support for static_key, static_jmp and static_call inside generated BPF trampoline. Please see patch 3 for details. [1] https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git/commit/?h=ftrace/direct&id=3ac423d902727884a389699fd7294c0e2e94b29c [2] https://lore.kernel.org/lkml/20191007082708.01393931.1@xxxxxxxxxxxxx/ Alexei Starovoitov (7): bpf, ftrace: temporary workaround bpf: refactor x86 JIT into helpers bpf: Introduce BPF trampoline libbpf: Add support to attach to fentry/fexit tracing progs selftest/bpf: Simple test for fentry/fexit bpf: Add kernel test functions for fentry testing selftests/bpf: Add test for BPF trampoline arch/x86/kernel/ftrace.c | 36 ++ arch/x86/net/bpf_jit_comp.c | 352 +++++++++++++++--- include/linux/bpf.h | 90 +++++ include/uapi/linux/bpf.h | 2 + kernel/bpf/Makefile | 1 + kernel/bpf/btf.c | 73 +++- kernel/bpf/core.c | 31 ++ kernel/bpf/syscall.c | 53 ++- kernel/bpf/trampoline.c | 252 +++++++++++++ kernel/bpf/verifier.c | 39 ++ net/bpf/test_run.c | 41 ++ tools/include/uapi/linux/bpf.h | 2 + tools/lib/bpf/bpf_helpers.h | 13 + tools/lib/bpf/libbpf.c | 55 ++- tools/lib/bpf/libbpf.h | 2 + tools/lib/bpf/libbpf.map | 1 + .../selftests/bpf/prog_tests/fentry_test.c | 65 ++++ .../selftests/bpf/prog_tests/kfree_skb.c | 37 +- .../testing/selftests/bpf/progs/fentry_test.c | 90 +++++ tools/testing/selftests/bpf/progs/kfree_skb.c | 52 +++ 20 files changed, 1215 insertions(+), 72 deletions(-) create mode 100644 kernel/bpf/trampoline.c create mode 100644 tools/testing/selftests/bpf/prog_tests/fentry_test.c create mode 100644 tools/testing/selftests/bpf/progs/fentry_test.c -- 2.23.0
