> -----Original Message----- > From: Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx> > Sent: Wednesday, December 15, 2021 6:11 PM > To: Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Jesper Dangaard > Brouer <jbrouer@xxxxxxxxxx>; Björn Töpel <bjorn@xxxxxxxxxx> > Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp <xdp- > newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong <boon.leong.ong@xxxxxxxxx>; > Joao Pedro Barros Silva <jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima > <dioli@xxxxxxxxxx> > Subject: Re: AF_XDP not transmitting frames immediately > > > > On 15/12/2021 12.07, Karlsson, Magnus wrote: > > > >> From: Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx> On 14/12/2021 > >> 09.07, Karlsson, Magnus wrote: > >>> > >>>> > >>>> I'm coding on an AF_XDP program[1] that need to send (a bulk of > >>>> packets) in a short time-window (related to Time-Triggered > >>>> Ethernet). > >>>> > >> [...] > >>> > >>>> How can I get AF_XDP to "flush" TX packets when calling sendto()? > >>>> Should we add another flag than the current MSG_DONTWAIT? > >>> > >>> In zero-copy mode with softirq driver processing (not busy poll), a > >>> sendto will just trigger the xsk_wakeup ndo that schedules napi > >>> unless it is already executing. It is up to the driver to then get > >>> packets from the Tx ring and put them on the HW and make sure they > >>> are sent. Barring any HW quirks, sending one packets should be > >>> perfectly fine. > >> > >> This actually doesn't sound so good from my customers use-case PoV. > >> That we only trigger a ndo_xsk_wakeup that schedules napi. > >> > >> We want to trigger HW transmission immediately. Can we achieve this > >> via using busy-poll mode? > > > > Yes, but not without napi. The napi context will in this case be > > executed in process context right away, unless it is already running > > somewhere else but that should not be the case. Will this be good > > enough? > > "Time" will tell if it is good enough (pun intended). > Meaning I will implement and measure it. > The busy-poll mode does sound like a way forward. > > Looking at kernel code, I can see that drivers TX NAPI usually does DMA-TX > completion *before* transmitting new frames. This usually makes sense, > but for our use-case of hitting a narrow time-slot, I worry about the jitter this > introduces. I would like to see a mode/flag that would allow transmitting > new frames (and afterwards invoking/scheduling TX-completion, e.g. via > raising the softirq/NAPI). > Well this is future work, first I will measure current implementation. Maciej has been experimenting with the ice driver to do sending first. Completions are only done lazily when needed to make sure that there are always a number of descriptors available for sending. This yields much better throughput and is the style that DPDK uses for its drivers. Hopefully, this will also improve latency, though we have not measured that. Could be adopted for the igc driver too if that is the case. > --Jesper
|