Hello, I'm seeing some extremely odd results with XDP once I change a parameter on the Mellanox ConnectX-4. The parameter is RX CQE moderation (rx_cqe_moder) in the NIC's priv flags, which is set to 'on' by default. When it is on, the system can XDP_DROP 87.8 MPPS (bottlenecked by PCI Bus) with about 88% cpu utilization (normalized across the network active cores). However, the weirdness happens when I turn this setting 'off'. Now, it can XDP_DROP 84.4 MPPS but the CPU util. is astoundingly low: around 1% (normalized across the network active cores). The drop in packets per second makes sense, as the RX CQE moderation should change the PCI behavior and as a result the PCI bottleneck. But does it make sense we can drop ~84 MPPS with extremely low CPU utilization across the network active CPUs? I ran an experiment to specifically probe the behavior when RX CQE moderation is 'off'. I varied the number of cores being assigned network traffic, and also added RX CQE compression as a factor to see if the behavior still existed at higher packet rates. I have attached graphs of the XDP drop rate and the CPU utilization. We can see in both cases (whether compression is on or off) that there is a steep drop off in CPU utilization as the XDP_DROP performance approaches the PCI bottleneck and we further increase the number of cores. I tried to get some intuition about what happens as the CPU utilization drops, but nothing jumps out at me. I used perf to see if there was a difference in cache (L1 and LLC) or TLB performance, but they were pretty consistent. I also generated perf flamegraphs for these cases, but I couldn't spot any interesting differences in behavior. Any ideas why this is happening? Thanks, Adam
<<attachment: DPDK_XDP_drop_util.pdf>>
<<attachment: DPDK_XDP_drop_perf.pdf>>
|