On Tue, Dec 14, 2021 at 10:40:05AM +0000, Karlsson, Magnus wrote: > Adding Ederson and Maciej. > > > On 14/12/2021 09.07, Karlsson, Magnus wrote: > > > > > > > > >> -----Original Message----- From: Jesper Dangaard Brouer > > >> <jbrouer@xxxxxxxxxx> Sent: Monday, December 13, 2021 10:04 PM To: > > >> Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel > > >> <bjorn@xxxxxxxxxx> Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp > > >> <xdp- newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong > > >> <boon.leong.ong@xxxxxxxxx>; Joao Pedro Barros Silva > > >> <jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima <dioli@xxxxxxxxxx> > > >> Subject: AF_XDP not transmitting frames immediately > > >> > > >> Hi Magnus and Bjørn, > > >> > > >> I'm coding on an AF_XDP program[1] that need to send (a bulk of > > >> packets) in a short time-window (related to Time-Triggered Ethernet). > > >> > > >> My observations are that AF_XDP doesn't send the frames immediately. > > >> And yes, I do call sendto() to trigger a TX kick. In zero-copy mode > > >> this is particular bad. My program want to send 4 packets in a > > >> burst, but I'm observing 8 packets grouped together on the receiving > > >> host. > > >> > > >> Is the a known property of AF_XDP? > > > > > > Nope! It is supposed to be able to send one packet at a time, though I > > > have several times seen bugs in the drivers where the batching > > > behavior shines through like this, and once a bug in the core code. > > > There is even a test these days for just sending a single packet, > > > > Where is that test in the kernel tree? > > In tools/testing/selftests/bpf/xdpxceiver.c. It is the RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates on veth currently. I'd say it's driver's fault. Magnus fixed something similar for i40e: https://lore.kernel.org/netdev/20210401172107.1191618-3-anthony.l.nguyen@xxxxxxxxx/ We don't have currently igc HW on our side to dig this :< > > > > since we have had issues with this in the past. That test does pass in > > > bpf-next, but it is only run with the veth driver that does not > > > support zero-copy so could still be an issue. What driver are you > > > using in zero-copy mode and what kernel version are you on? > > > > Driver: igc with Intel chip i225 > > Have never tried this one personally. Do not know if I have one in the lab but let me check. > > Ederson, do you have any experience with this card and if so, have you seen something similar? > > > Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT > > - Devel branch at commit 6d3b1b069946 (v5.15-12802-g6d3b1b069946) > > > > >> How can I get AF_XDP to "flush" TX packets when calling sendto()? > > >> Should we add another flag than the current MSG_DONTWAIT? > > > > > > In zero-copy mode with softirq driver processing (not busy poll), a > > > sendto will just trigger the xsk_wakeup ndo that schedules napi unless > > > it is already executing. It is up to the driver to then get packets > > > from the Tx ring and put them on the HW and make sure they are sent. > > > Barring any HW quirks, sending one packets should be perfectly fine. > > > > I will investigate driver level issues. > > > > I have other (100G) NICs in my testlab, but I'm using these 1G NICs because > > they support hardware timestamping, which allows me to investigate these > > timing issues. > > I'll find a way to see of other drivers behave differently. > > Would be great if you could check if the problem also exists on e.g. ice. > > > >> Hint, I'm using tcpdump hardware timestamping on receiving hist via > > >> cmdline: > > >> > > >> tcpdump -vv -s0 -ni eth1 -j adapter_unsynced > > >> --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42 > > >> > > >> Notice[1] on specific branch: [1] > > >> https://github.com/xdp-project/bpf- > > examples/tree/vestas03_AF_XDP_exam > > >> ple/AF_XDP-interaction > > > > > > > Thanks for your feedback, > > --Jesper >
|