On 14/12/2021 12.25, Maciej Fijalkowski wrote:
On Tue, Dec 14, 2021 at 10:40:05AM +0000, Karlsson, Magnus wrote:Adding Ederson and Maciej.On 14/12/2021 09.07, Karlsson, Magnus wrote:-----Original Message----- From: Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx> Sent: Monday, December 13, 2021 10:04 PM To: Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel <bjorn@xxxxxxxxxx> Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp <xdp- newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong <boon.leong.ong@xxxxxxxxx>; Joao Pedro Barros Silva <jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima <dioli@xxxxxxxxxx> Subject: AF_XDP not transmitting frames immediately Hi Magnus and Bjørn, I'm coding on an AF_XDP program[1] that need to send (a bulk of packets) in a short time-window (related to Time-Triggered Ethernet). My observations are that AF_XDP doesn't send the frames immediately. And yes, I do call sendto() to trigger a TX kick. In zero-copy mode this is particular bad. My program want to send 4 packets in a burst, but I'm observing 8 packets grouped together on the receiving host. Is the a known property of AF_XDP?Nope! It is supposed to be able to send one packet at a time, though I have several times seen bugs in the drivers where the batching behavior shines through like this, and once a bug in the core code. There is even a test these days for just sending a single packet,Where is that test in the kernel tree?In tools/testing/selftests/bpf/xdpxceiver.c. It is the RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates on veth currently.I'd say it's driver's fault. Magnus fixed something similar for i40e: https://lore.kernel.org/netdev/20210401172107.1191618-3-anthony.l.nguyen@xxxxxxxxx/
Thanks for that hint.
We don't have currently igc HW on our side to dig this :<
I suspected Boon Leong (cc) would have this hardware.
since we have had issues with this in the past. That test does pass in bpf-next, but it is only run with the veth driver that does not support zero-copy so could still be an issue. What driver are you using in zero-copy mode and what kernel version are you on?Driver: igc with Intel chip i225Have never tried this one personally. Do not know if I have one in the lab but let me check. Ederson, do you have any experience with this card and if so, have you seen something similar?Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT - Devel branch at commit 6d3b1b069946 (v5.15-12802-g6d3b1b069946)How can I get AF_XDP to "flush" TX packets when calling sendto()? Should we add another flag than the current MSG_DONTWAIT?In zero-copy mode with softirq driver processing (not busy poll), a sendto will just trigger the xsk_wakeup ndo that schedules napi unless it is already executing. It is up to the driver to then get packets from the Tx ring and put them on the HW and make sure they are sent. Barring any HW quirks, sending one packets should be perfectly fine.I will investigate driver level issues. I have other (100G) NICs in my testlab, but I'm using these 1G NICs because they support hardware timestamping, which allows me to investigate these timing issues. I'll find a way to see of other drivers behave differently.Would be great if you could check if the problem also exists on e.g. ice.
Having issues getting my ICE hardware to link up.I tested that driver i40e works as expected. Thus, this is likely an issue with the driver. I will digg some more.
Hint, I'm using tcpdump hardware timestamping on receiving hist via cmdline: tcpdump -vv -s0 -ni eth1 -j adapter_unsynced --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42Notice[1] on specific branch:
[1] https://github.com/xdp-project/bpf-examples/tree/vestas03_AF_XDP_example/AF_XDP-interaction
In [1] I tried to play with SO_PREFER_BUSY_POLL, but it didn't make a difference.
[2] https://github.com/xdp-project/bpf-examples/commit/3685d5ea93fced --Jesper
|