Last week, I proposed non-intrusive BPF runtime hooks for implementing automatic resource release in watchdog [0]. I realized this weekend that the idea of non-intrusive BPF runtime hooks can have more application scenarios, and that it is a general infrastructure. For example, we can implement BPF version of strace based on BPF runtime hooks to help us diagnose and debug bpf programs. We can add BPF debug mode. When a bpf program is in debug mode, all calls to kfuncs and helpers will be traced and output to the trace ring buffer. We can also record timestamps for each call to analyze the performance of a bpf program, making it easier for us to optimize the performance of a bpf program. This patch series is a proof-of-concept implementation of BPF version of strace based on BPF runtime hooks. Note that this patch series is developed based on BPF runtime hooks and therefore cannot be applied separately. In this patch series I used trace_printk directly in the BPF debug mode hook for proof-of-concept purposes. In actual implementation, the BPF debug mode hook should only be responsible for recording information, and parsing and outputting information should be done in another thread to avoid affecting the performance of the bpf program. We may also need to add a new FTRACE_ENTRY to customize the format of the event output. We always use bpf program to trace/analyze/diagnose kernel/applications, and now we can also trace/analyze/diagnose bpf programs via BPF debug mode. This will be helpful for us to debug/optimize complex bpf programs. (An interesting idea is that maybe one day we will also be able to attach bpf programs to the BPF debug mode hook. We use a bpf program to trace other bpf programs in the kernel to help us improve the performance of other bpf programs. Of course, the bpf program attached to the debug mode hook cannot enter debug mode.) [0]: https://lore.kernel.org/bpf/AM6PR03MB5080513BFAEB54A93CC70D4399FE2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/T/#u The following is a simple test result (removed trace_printk prefix): [456.900891] bpf_iter_num_new(ffffa91a02103c78,1,3) = 0 [456.903768] bpf_iter_num_next(ffffa91a02103c78) = ffffa91a02103c78 [456.904849] bpf_task_from_pid(1) = ffff8bf4c1320000 [456.905904] bpf_probe_read_kernel(ffffa91a02103c74,4,ffff8bf4c1320690) = 0 [456.905971] bpf_task_release(ffff8bf4c1320000) [456.906893] bpf_iter_num_next(ffffa91a02103c78) = ffffa91a02103c78 [456.907751] bpf_task_from_pid(2) = ffff8bf4c1320fc0 [456.908635] bpf_probe_read_kernel(ffffa91a02103c74,4,ffff8bf4c1321650) = 0 [456.908656] bpf_task_release(ffff8bf4c1320fc0) [456.909570] bpf_iter_num_next(ffffa91a02103c78) = 0 [456.910425] bpf_iter_num_destroy(ffffa91a02103c78) [456.911604] bpf_cpumask_create() = ffff8bf4c4042a88 [456.912600] bpf_cpumask_release(ffff8bf4c4042a88) Any feedback is welcome. Many thanks. Signed-off-by: Juntong Deng <juntong.deng@xxxxxxxxxxx> Juntong Deng (3): bpf: Add BPF debug mode bpf: Add bpf_runtime_kfunc_tracing_hook selftests/bpf: Add test case for demonstrating BPF debug mode. arch/x86/net/bpf_jit_comp.c | 2 +- include/linux/bpf.h | 3 +- include/linux/btf.h | 4 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/btf.c | 72 +++++++++- kernel/bpf/syscall.c | 4 +- tools/include/uapi/linux/bpf.h | 2 + tools/lib/bpf/libbpf.c | 6 + tools/lib/bpf/libbpf.h | 2 + tools/testing/selftests/bpf_debug/Makefile | 136 ++++++++++++++++++ tools/testing/selftests/bpf_debug/debug.bpf.c | 39 +++++ tools/testing/selftests/bpf_debug/debug.c | 24 ++++ 12 files changed, 290 insertions(+), 5 deletions(-) create mode 100644 tools/testing/selftests/bpf_debug/Makefile create mode 100644 tools/testing/selftests/bpf_debug/debug.bpf.c create mode 100644 tools/testing/selftests/bpf_debug/debug.c -- 2.39.5
