On 6/19/24 9:17 PM, Toke Høiland-Jørgensen wrote:
Jesper Dangaard Brouer <hawk@xxxxxxxxxx> writes:
On 18/06/2024 17.28, Sebastiano Miano wrote:
I have been conducting some basic experiments with XDP and have
observed a significant performance regression in recent kernel
versions compared to v5.15.
My setup is the following:
- Hardware: Two machines connected back-to-back with 100G Mellanox
ConnectX-6 Dx.
- DUT: 2x16 core Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz.
- Software: xdp-bench program from [1] running on the DUT in both DROP
and TX modes.
- Traffic generator: Pktgen-DPDK sending traffic with a single 64B UDP
flow at ~130Mpps.
- Tests: Single core, HT disabled
Results:
Kernel version |-------| XDP_DROP |--------| XDP_TX |
5.15 30Mpps 16.1Mpps
6.2 21.3Mpps 14.1Mpps
6.5 19.9Mpps 8.6Mpps
bpf-next (6.10-rc2) 22.1Mpps 9.2Mpps
Around when I left Red Hat there were a project with [LNST] that used
xdp-bench for tracking and finding regressions like this.
Perhaps Toke can enlighten us, if that project have caught similar
regressions?
[LNST] https://github.com/LNST-project/lnst
Yes, actually, we have! Here's the bugzilla for it:
https://bugzilla.redhat.com/show_bug.cgi?id=2270408
> We compared performance of ELN and RHEL9 candidate kernels and noticed significant
> drop in XDP drop [1] on mlx5 (25G).
>
> On any rhel9 candidate kernel we are able to drop 19-20M pkts/sec but on an ELN
> kernels, we are reaching just 15M pkts/sec (CPU utillization remains the same -
> around 100%).
>
> We don't see such regression on ixgbe or i40e.
It looks like this is known since March, was this ever reported to Nvidia back
then? :/
Given XDP is in the critical path for many in production, we should think about
regular performance reporting for the different vendors for each released kernel,
similar to here [0].
Out of curiosity, @Saeed: Is Nvidia internally regularly assessing XDP perf for mlx5
as part of QA? (Probably not, but I thought I'd ask.)
Thanks,
Daniel
[0] http://core.dpdk.org/perf-reports/
