RE: AF_XDP not transmitting frames immediately

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




> -----Original Message-----
> From: Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx>
> Sent: Monday, December 13, 2021 10:04 PM
> To: Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel
> <bjorn@xxxxxxxxxx>
> Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp <xdp-
> newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong <boon.leong.ong@xxxxxxxxx>;
> Joao Pedro Barros Silva <jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima
> <dioli@xxxxxxxxxx>
> Subject: AF_XDP not transmitting frames immediately
> 
> Hi Magnus and Bjørn,
> 
> I'm coding on an AF_XDP program[1] that need to send (a bulk of packets) in
> a short time-window (related to Time-Triggered Ethernet).
> 
> My observations are that AF_XDP doesn't send the frames immediately.
> And yes, I do call sendto() to trigger a TX kick.
> In zero-copy mode this is particular bad.  My program want to send 4 packets
> in a burst, but I'm observing 8 packets grouped together on the receiving
> host.
> 
> Is the a known property of AF_XDP?

Nope! It is supposed to be able to send one packet at a time, though I have several times seen bugs in the drivers where the batching behavior shines through like this, and once a bug in the core code. There is even a test these days for just sending a single packet, since we have had issues with this in the past. That test does pass in bpf-next, but it is only run with the veth driver that does not support zero-copy so could still be an issue. What driver are you using in zero-copy mode and what kernel version are you on?

> How can I get AF_XDP to "flush" TX packets when calling sendto()?
> Should we add another flag than the current MSG_DONTWAIT?

In zero-copy mode with softirq driver processing (not busy poll), a sendto will just trigger the xsk_wakeup ndo that schedules napi unless it is already executing. It is up to the driver to then get packets from the Tx ring and put them on the HW and make sure they are sent. Barring any HW quirks, sending one packets should be perfectly fine.

/Magnus

> --Jesper
> 
> Hint, I'm using tcpdump hardware timestamping on receiving hist via cmdline:
> 
>   tcpdump -vv -s0 -ni eth1 -j adapter_unsynced --time-stamp-precision=nano
> -w af_xdp_tx_cyclic.dump42
> 
> Notice[1] on specific branch:
>   [1]
> https://github.com/xdp-project/bpf-
> examples/tree/vestas03_AF_XDP_example/AF_XDP-interaction





[Index of Archives]     [Linux Networking Development]     [Fedora Linux Users]     [Linux SCTP]     [DCCP]     [Gimp]     [Yosemite Campsites]

  Powered by Linux