On 14/12/2021 09.07, Karlsson, Magnus wrote:
-----Original Message----- From: Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx> Sent: Monday, December 13, 2021 10:04 PM To:Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel <bjorn@xxxxxxxxxx> Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp<xdp- newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong <boon.leong.ong@xxxxxxxxx>; Joao Pedro Barros Silva<jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima <dioli@xxxxxxxxxx> Subject: AF_XDP not transmitting framesimmediately Hi Magnus and Bjørn, I'm coding on an AF_XDP program[1] that need to send (a bulk of packets) in a short time-window (related to Time-Triggered Ethernet). My observations are that AF_XDP doesn't send the frames immediately. And yes, I do call sendto() to trigger a TX kick. In zero-copy mode this is particular bad. My program want to send 4 packets in a burst, but I'm observing 8 packets grouped together on the receiving host. Is the a known property of AF_XDP?Nope! It is supposed to be able to send one packet at a time, though I have several times seen bugs in the drivers where the batching behavior shines through like this, and once a bug in the core code. There is even a test these days for just sending a single packet,
Where is that test in the kernel tree?
since we have had issues with this in the past. That test does pass in bpf-next, but it is only run with the veth driver that does not support zero-copy so could still be an issue. What driver are you using in zero-copy mode and what kernel version are you on?
Driver: igc with Intel chip i225 Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT - Devel branch at commit 6d3b1b069946 (v5.15-12802-g6d3b1b069946)
How can I get AF_XDP to "flush" TX packets when calling sendto()? Should we add another flag than the current MSG_DONTWAIT?In zero-copy mode with softirq driver processing (not busy poll), a sendto will just trigger the xsk_wakeup ndo that schedules napi unless it is already executing. It is up to the driver to then get packets from the Tx ring and put them on the HW and make sure they are sent. Barring any HW quirks, sending one packets should be perfectly fine.
I will investigate driver level issues.I have other (100G) NICs in my testlab, but I'm using these 1G NICs because they support hardware timestamping, which allows me to investigate these timing issues.
I'll find a way to see of other drivers behave differently.
Hint, I'm using tcpdump hardware timestamping on receiving hist via cmdline: tcpdump -vv -s0 -ni eth1 -j adapter_unsynced --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42Notice[1] on specific branch: [1] https://github.com/xdp-project/bpf-examples/tree/vestas03_AF_XDP_example/AF_XDP-interaction
Thanks for your feedback, --Jesper
|