Re: Bad XDP performance with mlx5

Tom Barbette <barbette@xxxxxx> · Wed, 29 May 2019 20:16:46 +0200

On 2019-05-29 19:16, Jesper Dangaard Brouer wrote:

On Wed, 29 May 2019 18:03:08 +0200
Tom Barbette <barbette@xxxxxx> wrote:

Hi all,

I've got a very simple eBPF program that counts packets per queue in a
per-cpu map.

Like xdp_rxq_info --dev mlx5p1 --action XDP_PASS ?

Even simpler.

I use IPerf in TCP mode, I limit the CPU cores to 2 so performance is
limited by CPU (always at 100%).

With a XL710 NIC 40G link, with the XDP program loaded, I get 32.5.
Without I get ~33.3Gbps. Pretty similar, somehow expected.

With a ConnectX 5 100G link, I get ~33.3Gbps without the XDP program but
~26 with it. The behavior seems similar with a simple XDP_PASS program.

Are you sure?

xdp_pass.c:
---
#include <linux/bpf.h>

#ifndef __section
# define __section(NAME)                  \
   __attribute__((section(NAME), used))
#endif

__section("prog")
int xdp_drop(struct xdp_md *ctx) {
    return XDP_PASS;
}

char __license[] __section("license") = "GPL";
---
clang -O2 -target bpf -c xdp_pass.c -o xdp_pass.o

Then see results with netperf below.

My test on a ConnectX 5 100G link show:
  - 33.8 Gbits/sec = with no-XDP prog
  - 34.5 Gbits/sec - with xdp_rxq_info

Even faster? :p

Any idea why MLX5 driver behaves like this? perf top is not conclusive
at first glance. I'd say check_object_size and
copy_user_enhanced_fast_string rise up but the stack is unclear from where.

It is possible to get very different and varying TCP bandwidth results,

depending on if TCP-server-process is running on the same CPU as the
NAPI-RX loop.  If they share the CPU then results are worse, as
process-context scheduling is setting a limit.

IPerf has one instance per-core, with SO_REUSEPORT and a BPF filter to 

map queues <-> CPU in 1:1 with irqbalance killed and set_affinity*sh.

So the setup on that regard is similar between tests and the variance do 

not come from different assignments.

Which is not what you're advising but ensure a similar per-core 

"pipeline" and tests reproducibility. It's a side question but any link 

on this L1/L2 cache misses vs scheduling question is welcome.

This is easiest to demonstrate with netperf option -Tn,n:

$ netperf -H 198.18.1.1 -D1 -T2,2 -t TCP_STREAM -l 120
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 198.18.1.1 () port 0 AF_INET : histogram : demo : cpu bind
Interim result: 35344.39 10^6bits/s over 1.002 seconds ending at 1559149724.219
Interim result: 35294.66 10^6bits/s over 1.001 seconds ending at 1559149725.221
Interim result: 36112.09 10^6bits/s over 1.002 seconds ending at 1559149726.222
Interim result: 36301.13 10^6bits/s over 1.000 seconds ending at 1559149727.222
^CInterim result: 36146.78 10^6bits/s over 0.507 seconds ending at 1559149727.730
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

131072  16384  16384    4.51     35801.94

server$ sudo service netperf start
server$ sudo killall -9 irqbalance
server$ sudo ethtool -X dpdk1 equal 2
server$ sudo ip link set dev dpdk1 xdp off
client$ netperf -H 10.220.0.5 -D1 -T2,2 -t TCP_STREAM -l 120

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 

10.220.0.5 () port 0 AF_INET : demo : cpu bind

Interim result: 37221.90 10^6bits/s over 1.015 seconds ending at 

1559151699.433

Interim result: 37811.52 10^6bits/s over 1.003 seconds ending at 

1559151700.436

Interim result: 38195.47 10^6bits/s over 1.001 seconds ending at 

1559151701.437

Interim result: 41089.18 10^6bits/s over 1.000 seconds ending at 

1559151702.437

Interim result: 38005.40 10^6bits/s over 1.081 seconds ending at 

1559151703.518

Interim result: 34419.33 10^6bits/s over 1.104 seconds ending at 

1559151704.622

^CInterim result: 40634.33 10^6bits/s over 0.198 seconds ending at 

1559151704.820