On Wed, Mar 25, 2020 at 11:24 AM Karlsson, Magnus
<magnus.karlsson@xxxxxxxxx> wrote:
> On Wed, Mar 25, 2020 at 11:02 AM Gaul, Maximilian
> <maximilian.gaul@xxxxxx> wrote:
> >
> > On Wed, Mar 25, 2020 at 10:41 AM Karlsson, Magnus
> > <magnus.karlsson@xxxxxxxxx> wrote:
> >
> > > On Wed, Mar 25, 2020 at 10:04 AM Gaul, Maximilian
> > > <maximilian.gaul@xxxxxx> wrote:
> > > >
> > > > I am running a Multi-AF-XDP-Socket approach per RX-Queue (using Shared Umem).
> > > >
> > > > Unfortunately I am noticing, that at around 650k pps, the *ksoftirqd*-thread of that RX-Queue ramps up to 100% thus leading to packet loss.
> > > > I tried setting *XDP_USE_NEED_WAKEUP* on *xsk_socket_cfg.bind_flags* but those bind_flags are only taken into account if *umem->refcount > 1* (libbpf/xsk.c - xsk_socket__create()).
> > > > As far as I understand this correctly, only the first socket is able to set *XDP_USE_NEED_WAKEUP* because for all sockets after, *umem->refcount* is going to be at least 2.
> > >
> > > Yes, the other sockets just inherit the settings of the first one.
> > >
> > > Are you using the SKB mode? What is your packet size? Sounds like a
> > > low number unless you have large packets and are using the SKB mode.
> > >
> >
> > These are the flags I set right before calling `xsk_socket__create`:
> >
> > xsk_socket_cfg.xdp_flags = cfg->xdp_flags | XDP_FLAGS_DRV_MODE | XDP_ZEROCOPY;
> > xsk_socket_cfg.bind_flags = cfg->xsk_bind_flags | XDP_USE_NEED_WAKEUP;
>
> XDP_ZEROCOPY is a bind flag not an XDP flag, so please move it there.
> If you get an error when you have it set, it means that your setup
> does not support zero-copy for some reason. Check what kernel version
> you are using and the the driver you are using supports zero-copy. I
> believe you need to use a queue id>=32 in the Mellanox driver for it
> to work in zero-copy mode. Below 32, you will get copy mode.
>
> > Packet size is around 1492 bytes.
>
> Seems that you are using SKB mode then, not zero-copy.
>
> /Magnus
>
Thank you for the hint. As you correctly said, I get an error if I use *XDP_ZEROCOPY*. But as far as I understand, packet rates should be higher in Driver-Mode even without zero-copy?
I just updated to the latest driver- and firmware version:
$ sudo ethtool -i <if>
driver: mlx5_core
version: 5.0-0
firmware-version: 16.27.1016 (MT_0000000012)
I actually have to correct myself: Incomming packets are 1442 bytes.
Can you give me the link between packet size and whether the NIC is running in SKB or DRV mode?
Mr. Brouer held a talk about (https://people.netfilter.org/hawk/presentations/driving-IT2017/driving-IT-2017_XDP_eBPF_technology_Jesper_Brouer.pdf) about XDP, mentioning in slide 11/27 that *mlx5 (v4.9)* has native XDP support.
> > Just to make sure: Those 650k packets are arriving on the same RX-Queue (even though this NIC has multiple RX-Queues I want to test maximum bandwith for a single RX-Queue).
> >
> > > > I didn't observe a dramatic change as I've hoped to. Are there some other ways to reduce interrupt load (user-space application and ksoftirq are already running on different CPUs)?
> > >
> > > The need_wakeup flag has a big impact when you run the softirq and the
> > > application thread on the same core. When using two cores for this, it
> > > has less of an impact.
> > >
> > > /Magnus
> > >
> > > > NIC: Mellanox Technologies MT27800
> > > >
> > > > Best regards
> > > >
> > > > Max
> > > >
> >
|