On 02/08/2023 14:52, 刘畅 wrote: > Hi all, > > I need to send trace data from in-kernel eBPF programs to user space. I have two choices, the first is storing traces in a BPF_MAP_TYPE_ARRAY map, and the user space consumer pulls traces from it. The second is storing traces in a BPF_MAP_TYPE_PROG_ARRAY map using bpf_perf_event_output(), and the user space consumer reads traces in the callback function invoked by perf_buffer__poll(). I did simple performance tests and found the two methods have significant performance difference. > > For the first method (pulling from a BPF_MAP_TYPE_ARRAY map), I set up an array map of 65536 entries and 40 bytes value size, then pull entries in user spcae for 10M times, and it costs 3.7s in average. > > For the second method, I allocate 2MB memory for the perf buffer of each CPU. The user space consumer calls perf_buffer__poll() in an infinite loop. To generate enough traces, I attach an ebpf program at the sys_enter_read tracepoint which will generate 100 traces in an execution, and run a user space program to call the read() system call in an infinite loop to trigger the ebpf program. The result is, it takes 10+ seconds to get 10M traces using perf_buffer__poll(), which is much slower than polling the array. > > This blog (https://nakryiko.com/posts/bpf-ringbuf/) says that bpf perf buffer has the ability to efficiently read data from user-space through memory-mapped region without extra memory copying and/or syscalls into the kernel, so I though it would be faster than reading the array map, which needs to invoke the bpf() system call. But my test gives the opposite result. I run this test on a server with 48 CPU cores and 188GB memory. The OS is Ubuntu 20.04 with kernel version 5.4.0. I wonder is this result as expected, or did I overlook something? > If you look in detail at the post you've referenced, it talks about the BPF ring buffer, which is different from BPF perf buffer. There's a guide there to converting BPF perf buffer API usage to BPF ring buffer usage - I'd suggest retrying using BPF ring buffer if your kernel is newer than 5.8. Alan > Thank you for your help! > > Chang Liu > Tsinghua University, China
