Re: AF_XDP not transmitting frames immediately

[Jesper Dangaard Brouer <jbrouer@xxxxxxxxxx>]

On 15/12/2021 02.08, Desouza, Ederson wrote:
> +Vinicius
>
> On Tue, 2021-12-14 at 10:40 +0000, Karlsson, Magnus wrote:
> > Adding Ederson and Maciej.
> >
> > > On 14/12/2021 09.07, Karlsson, Magnus wrote:
> > > > > From: Jesper Dangaard Brouer 
> > > > >
> > > > > Hi Magnus and Bjørn,
> > > > >
> > > > > I'm coding on an AF_XDP program[1] that need to send (a bulk
> > > > > of packets) in a short time-window (related to
> > > > > Time-Triggered Ethernet).
> > > > >
> > > > > My observations are that AF_XDP doesn't send the frames 
> > > > > immediately. And yes, I do call sendto() to trigger a TX
> > > > > kick. In zero-copy mode this is particular bad.  My program
> > > > > want to send 4 packets in a burst, but I'm observing 8
> > > > > packets grouped together on the receiving host.
> > > > >
> > > > > Is the a known property of AF_XDP?
> > > >
> > > > Nope! It is supposed to be able to send one packet at a time, 
> > > > though I have several times seen bugs in the drivers where the
> > > > batching behavior shines through like this, and once a bug in
> > > > the core code. There is even a test these days for just sending
> > > > a single packet,
> > >
> > > Where is that test in the kernel tree?
> >
> > In tools/testing/selftests/bpf/xdpxceiver.c. It is the 
> > RUN_TO_COMPLETION_SINGLE_PKT test. But the framework only operates
> > on veth currently.
> >
> > > > since we have had issues with this in the past. That test does 
> > > > pass in bpf-next, but it is only run with the veth driver that
> > > > does not support zero-copy so could still be an issue. What
> > > > driver are you using in zero-copy mode and what kernel version
> > > > are you on?
> > >
> > > Driver: igc with Intel chip i225
> >
> > Have never tried this one personally. Do not know if I have one in 
> > the lab but let me check.
> >
> > Ederson, do you have any experience with this card and if so, have 
> > you seen something similar?
>
> Not sure. I wonder how small is the interval Jesper is using. I
> imagine that if it's too small, the interrupt generated to trigger
> the tx could end up serving more than one packet.

The interval is currently 1 second (code here[0]), thus the TX interrupt 
should have plenty of time to trigger.

[0] 
https://github.com/xdp-project/bpf-examples/blob/vestas03_AF_XDP_example/AF_XDP-interaction/af_xdp_user.c#L1129


> Vinicius should have more prompt access to i225 - could you please
> help on this?

My reproducer[1] need option --zero-copy to enable the error case, as it 
defaults to 'copy-mode'.
It is only in zero-copy mode for igc/i225 that I see the behavior of 8 
packet bulking, when expecting/sending bulks of 4 packets ever second.


[1] 
https://github.com/xdp-project/bpf-example/tree/vestas03_AF_XDP_example/AF_XDP-interaction

--Jesper