On Wed, 23 Oct 2019 at 19:42, Alexei Starovoitov <alexei.starovoitov@xxxxxxxxx> wrote: > > On Tue, Oct 22, 2019 at 12:06 PM Samudrala, Sridhar > <sridhar.samudrala@xxxxxxxxx> wrote: > > > > OK. Here is another data point that shows the perf report with the same test but CPU mitigations > > turned OFF. Here bpf_prog overhead goes down from almost (10.18 + 4.51)% to (3.23 + 1.44%). > > > > 21.40% ksoftirqd/28 [i40e] [k] i40e_clean_rx_irq_zc > > 14.13% xdpsock [i40e] [k] i40e_clean_rx_irq_zc > > 8.33% ksoftirqd/28 [kernel.vmlinux] [k] xsk_rcv > > 6.09% ksoftirqd/28 [kernel.vmlinux] [k] xdp_do_redirect > > 5.19% xdpsock xdpsock [.] main > > 3.48% ksoftirqd/28 [kernel.vmlinux] [k] bpf_xdp_redirect_map > > 3.23% ksoftirqd/28 bpf_prog_3c8251c7e0fef8db [k] bpf_prog_3c8251c7e0fef8db > > > > So a major component of the bpf_prog overhead seems to be due to the CPU vulnerability mitigations. > > I feel that it's an incorrect conclusion because JIT is not doing > any retpolines (because there are no indirect calls in bpf). > There should be no difference in bpf_prog runtime with or without mitigations. > Also you're running root, so no spectre mitigations either. > > This 3% seems like a lot for a function that does few loads that should > hit d-cache and one direct call. > Please investigate why you're seeing this 10% cpu cost when mitigations are on. > perf report/annotate is the best. > Also please double check that you're using the latest perf. > Since bpf performance analysis was greatly improved several versions ago. > I don't think old perf will be showing bogus numbers, but better to > run the latest. > For comparison, on my Skylake 3GHz with mitigations off. (I have one internal patch that inlines xsk_rcv() into __xsk_map_redirect, so that's why it's not showing xsk_rcv(). I'll upstream that...) 41.79% [kernel] [k] i40e_clean_rx_irq_zc 15.55% [kernel] [k] __xsk_map_redirect 9.87% [kernel] [k] xdp_do_redirect 6.89% [kernel] [k] xsk_umem_peek_addr 6.37% [kernel] [k] bpf_xdp_redirect_map 5.02% bpf_prog_992d9ddc835e5629 [k] bpf_prog_992d9ddc835e5629 Again, it might look weird that simple functions like bpf_xdp_redirect_map and the XDP program is 6% and 5%. Let's dig into that. I let the xdpsock program (rxdrop) run on one core 22, and the ksoftirqd on core 20. Core 20 is only processing packets, plus the regular kernel householding. I did a processor trace for core 20 for 207 936 packets. In total it's 84,407,427 instructions, and bpf_xdp_redirect_map() is 8,109,504 instructions, which is 9.6%. bpf_xdp_redirect_map() executes 39 instructions for AF_XDP. As perf is reporting less than 9.6% means that the IPC count of that function is more than the average which perf-stat reports as IPC of 2.88. The BPF program executes fewer instructions than bpf_xdp_redirect_map(), so given that perf shows 5%, means that the IPC count is better than average here as well. So, it's roughly 405 instructions per packet, and with an IPC of 2.88 that'll give ~140 cycles per packet, which on this machine (3,000,000,000/140) is ~21.4 Mpps. The xdpsock application reports 21. This is sane. The TL;DR version is: 6% and 5% for bpf_xdp_redirect_map and bpf_prog_992d9ddc835e5629 might seem high, but it's just that the total number of instruction executing is fairly low. So, even though the functions are small in size, it will show up as non-negligible percentage in perf. At these speeds, really small things have an impact on performance. DPDK has ~50 cycles per packet. Björn > > The other component is the bpf_xdp_redirect_map() codepath. > > > > Let me know if it helps to collect any other data that should further help with the perf analysis. > >
