Adding Ederson and Maciej. > -----Original Message----- > From: Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx> > Sent: Tuesday, December 14, 2021 11:32 AM > To: Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel > <bjorn@xxxxxxxxxx> > Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp <xdp- > newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong <boon.leong.ong@xxxxxxxxx>; > Joao Pedro Barros Silva <jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima > <dioli@xxxxxxxxxx> > Subject: Re: AF_XDP not transmitting frames immediately > > > > On 14/12/2021 09.07, Karlsson, Magnus wrote: > > > > > >> -----Original Message----- From: Jesper Dangaard Brouer > >> <jbrouer@xxxxxxxxxx> Sent: Monday, December 13, 2021 10:04 PM To: > >> Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel > >> <bjorn@xxxxxxxxxx> Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp > >> <xdp- newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong > >> <boon.leong.ong@xxxxxxxxx>; Joao Pedro Barros Silva > >> <jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima <dioli@xxxxxxxxxx> > >> Subject: AF_XDP not transmitting frames immediately > >> > >> Hi Magnus and Bjørn, > >> > >> I'm coding on an AF_XDP program[1] that need to send (a bulk of > >> packets) in a short time-window (related to Time-Triggered Ethernet). > >> > >> My observations are that AF_XDP doesn't send the frames immediately. > >> And yes, I do call sendto() to trigger a TX kick. In zero-copy mode > >> this is particular bad. My program want to send 4 packets in a > >> burst, but I'm observing 8 packets grouped together on the receiving > >> host. > >> > >> Is the a known property of AF_XDP? > > > > Nope! It is supposed to be able to send one packet at a time, though I > > have several times seen bugs in the drivers where the batching > > behavior shines through like this, and once a bug in the core code. > > There is even a test these days for just sending a single packet, > > Where is that test in the kernel tree? In tools/testing/selftests/bpf/xdpxceiver.c. It is the RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates on veth currently. > > since we have had issues with this in the past. That test does pass in > > bpf-next, but it is only run with the veth driver that does not > > support zero-copy so could still be an issue. What driver are you > > using in zero-copy mode and what kernel version are you on? > > Driver: igc with Intel chip i225 Have never tried this one personally. Do not know if I have one in the lab but let me check. Ederson, do you have any experience with this card and if so, have you seen something similar? > Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT > - Devel branch at commit 6d3b1b069946 (v5.15-12802-g6d3b1b069946) > > >> How can I get AF_XDP to "flush" TX packets when calling sendto()? > >> Should we add another flag than the current MSG_DONTWAIT? > > > > In zero-copy mode with softirq driver processing (not busy poll), a > > sendto will just trigger the xsk_wakeup ndo that schedules napi unless > > it is already executing. It is up to the driver to then get packets > > from the Tx ring and put them on the HW and make sure they are sent. > > Barring any HW quirks, sending one packets should be perfectly fine. > > I will investigate driver level issues. > > I have other (100G) NICs in my testlab, but I'm using these 1G NICs because > they support hardware timestamping, which allows me to investigate these > timing issues. > I'll find a way to see of other drivers behave differently. Would be great if you could check if the problem also exists on e.g. ice. > >> Hint, I'm using tcpdump hardware timestamping on receiving hist via > >> cmdline: > >> > >> tcpdump -vv -s0 -ni eth1 -j adapter_unsynced > >> --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42 > >> > >> Notice[1] on specific branch: [1] > >> https://github.com/xdp-project/bpf- > examples/tree/vestas03_AF_XDP_exam > >> ple/AF_XDP-interaction > > > > Thanks for your feedback, > --Jesper
|