On Wed, Mar 25, 2020 at 10:41 AM Karlsson, Magnus
<magnus.karlsson@xxxxxxxxx> wrote:
> On Wed, Mar 25, 2020 at 10:04 AM Gaul, Maximilian
> <maximilian.gaul@xxxxxx> wrote:
> >
> > I am running a Multi-AF-XDP-Socket approach per RX-Queue (using Shared Umem).
> >
> > Unfortunately I am noticing, that at around 650k pps, the *ksoftirqd*-thread of that RX-Queue ramps up to 100% thus leading to packet loss.
> > I tried setting *XDP_USE_NEED_WAKEUP* on *xsk_socket_cfg.bind_flags* but those bind_flags are only taken into account if *umem->refcount > 1* (libbpf/xsk.c - xsk_socket__create()).
> > As far as I understand this correctly, only the first socket is able to set *XDP_USE_NEED_WAKEUP* because for all sockets after, *umem->refcount* is going to be at least 2.
>
> Yes, the other sockets just inherit the settings of the first one.
>
> Are you using the SKB mode? What is your packet size? Sounds like a
> low number unless you have large packets and are using the SKB mode.
>
These are the flags I set right before calling `xsk_socket__create`:
xsk_socket_cfg.xdp_flags = cfg->xdp_flags | XDP_FLAGS_DRV_MODE | XDP_ZEROCOPY;
xsk_socket_cfg.bind_flags = cfg->xsk_bind_flags | XDP_USE_NEED_WAKEUP;
Packet size is around 1492 bytes.
Just to make sure: Those 650k packets are arriving on the same RX-Queue (even though this NIC has multiple RX-Queues I want to test maximum bandwith for a single RX-Queue).
> > I didn't observe a dramatic change as I've hoped to. Are there some other ways to reduce interrupt load (user-space application and ksoftirq are already running on different CPUs)?
>
> The need_wakeup flag has a big impact when you run the softirq and the
> application thread on the same core. When using two cores for this, it
> has less of an impact.
>
> /Magnus
>
> > NIC: Mellanox Technologies MT27800
> >
> > Best regards
> >
> > Max
> >
|