+Vinicius On Tue, 2021-12-14 at 10:40 +0000, Karlsson, Magnus wrote: > Adding Ederson and Maciej. > > > -----Original Message----- > > From: Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx> > > Sent: Tuesday, December 14, 2021 11:32 AM > > To: Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel > > <bjorn@xxxxxxxxxx> > > Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp <xdp- > > newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong > > <boon.leong.ong@xxxxxxxxx>; > > Joao Pedro Barros Silva <jopbs@xxxxxxxxxx>; Diogo Alexandre Da > > Silva Lima > > <dioli@xxxxxxxxxx> > > Subject: Re: AF_XDP not transmitting frames immediately > > > > > > > > On 14/12/2021 09.07, Karlsson, Magnus wrote: > > > > > > > > > > -----Original Message----- From: Jesper Dangaard Brouer > > > > <jbrouer@xxxxxxxxxx> Sent: Monday, December 13, 2021 10:04 PM > > > > To: > > > > Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; Björn Töpel > > > > <bjorn@xxxxxxxxxx> Cc: Brouer, Jesper <brouer@xxxxxxxxxx>; Xdp > > > > <xdp- newbies@xxxxxxxxxxxxxxx>; Ong, Boon Leong > > > > <boon.leong.ong@xxxxxxxxx>; Joao Pedro Barros Silva > > > > <jopbs@xxxxxxxxxx>; Diogo Alexandre Da Silva Lima > > > > <dioli@xxxxxxxxxx> > > > > Subject: AF_XDP not transmitting frames immediately > > > > > > > > Hi Magnus and Bjørn, > > > > > > > > I'm coding on an AF_XDP program[1] that need to send (a bulk of > > > > packets) in a short time-window (related to Time-Triggered > > > > Ethernet). > > > > > > > > My observations are that AF_XDP doesn't send the frames > > > > immediately. > > > > And yes, I do call sendto() to trigger a TX kick. In zero-copy > > > > mode > > > > this is particular bad. My program want to send 4 packets in a > > > > burst, but I'm observing 8 packets grouped together on the > > > > receiving > > > > host. > > > > > > > > Is the a known property of AF_XDP? > > > > > > Nope! It is supposed to be able to send one packet at a time, > > > though I > > > have several times seen bugs in the drivers where the batching > > > behavior shines through like this, and once a bug in the core > > > code. > > > There is even a test these days for just sending a single packet, > > > > Where is that test in the kernel tree? > > In tools/testing/selftests/bpf/xdpxceiver.c. It is the > RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates on > veth currently. > > > > since we have had issues with this in the past. That test does > > > pass in > > > bpf-next, but it is only run with the veth driver that does not > > > support zero-copy so could still be an issue. What driver are you > > > using in zero-copy mode and what kernel version are you on? > > > > Driver: igc with Intel chip i225 > > Have never tried this one personally. Do not know if I have one in > the lab but let me check. > > Ederson, do you have any experience with this card and if so, have > you seen something similar? Not sure. I wonder how small is the interval Jesper is using. I imagine that if it's too small, the interrupt generated to trigger the tx could end up serving more than one packet. Vinicius should have more prompt access to i225 - could you please help on this? > > > Kernel version: 5.15.0-net-next+ #618 SMP PREEMPT > > - Devel branch at commit 6d3b1b069946 (v5.15-12802- > > g6d3b1b069946) > > > > > > How can I get AF_XDP to "flush" TX packets when calling > > > > sendto()? > > > > Should we add another flag than the current MSG_DONTWAIT? > > > > > > In zero-copy mode with softirq driver processing (not busy poll), > > > a > > > sendto will just trigger the xsk_wakeup ndo that schedules napi > > > unless > > > it is already executing. It is up to the driver to then get > > > packets > > > from the Tx ring and put them on the HW and make sure they are > > > sent. > > > Barring any HW quirks, sending one packets should be perfectly > > > fine. > > > > I will investigate driver level issues. > > > > I have other (100G) NICs in my testlab, but I'm using these 1G NICs > > because > > they support hardware timestamping, which allows me to investigate > > these > > timing issues. > > I'll find a way to see of other drivers behave differently. > > Would be great if you could check if the problem also exists on e.g. > ice. > > > > > Hint, I'm using tcpdump hardware timestamping on receiving hist > > > > via > > > > cmdline: > > > > > > > > tcpdump -vv -s0 -ni eth1 -j adapter_unsynced > > > > --time-stamp-precision=nano -w af_xdp_tx_cyclic.dump42 > > > > > > > > Notice[1] on specific branch: [1] > > > > https://github.com/xdp-project/bpf- > > examples/tree/vestas03_AF_XDP_exam > > > > ple/AF_XDP-interaction > > > > > > > Thanks for your feedback, > > --Jesper >
|