On Sat, 7 Oct 2023 at 01:07, Srivats P <pstavirs@xxxxxxxxx> wrote: > > On Fri, Oct 6, 2023 at 6:34 PM Magnus Karlsson > <magnus.karlsson@xxxxxxxxx> wrote: > > > > On Fri, 6 Oct 2023 at 14:36, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > > > On Fri, Oct 6, 2023 at 5:23 PM Magnus Karlsson > > > <magnus.karlsson@xxxxxxxxx> wrote: > > > > > > > > On Fri, 6 Oct 2023 at 13:41, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > > > > > > > On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson > > > > > <magnus.karlsson@xxxxxxxxx> wrote: > > > > > > > > > > > > On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > > > > > > > > > > > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson > > > > > > > <magnus.karlsson@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I want to clarify my AF_XDP understanding for a particular scenario. > > > > > > > > > > > > > > > > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and > > > > > > > > > single XDP socket bound to queue 0 with 1 Tx Ring but NO Rx Ring. > > > > > > > > > Assume the NIC has only a single queue (to keep things simple for > > > > > > > > > explaining this scenario). > > > > > > > > > > > > > > > > > > There is a XDP program attached to Queue 0 which does either a > > > > > > > > > XDP_DROP or XDP_PASS for all Rx packets. > > > > > > > > > > > > > > > > > > We are running in Driver mode. > > > > > > > > > > > > > > > > > > What happens to RX packets received on Queue 0? > > > > > > > > > > > > > > > > > > Here's my understanding - > > > > > > > > > > > > > > > > > > * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring > > > > > > > > > for queue 0 > > > > > > > > > * The NIC Rx Ring for queue 0 is populated by the driver with UMEM > > > > > > > > > buffers taken from the Fill ring > > > > > > > > > > > > > > > > In driver mode (i.e. not zero-copy mode) this will not happen. The > > > > > > > > ring will be populated by kernel buffers. > > > > > > > > > > > > > > What is the behaviour in zero-copy mode? > > > > > > > > > > > > Then the behaviour is according to what you wrote above. Just note > > > > > > that nothing will be returned in the fill ring as you are not sending > > > > > > anything to the AF_XDP socket. > > > > > > > > > > I assume you meant "Rx Ring" and not "Fill ring" above? > > > > > > > > Yes, sorry. Nothing will arrive in the Rx ring, and the Fill ring will > > > > not be used apart from the initial burst of buffers that will be > > > > grabbed by the system. Note though that you have to provide some > > > > buffers in the fill ring as these are the ones that are used by the > > > > NIC in zero-copy mode. They are just never returned to user-space > > > > through the Rx ring. They are just recycled in the driver so the > > > > system never needs to grab more in your scenario. > > > > > > > > > > After a XDP_DROP or XDP_PASS at RX, will the driver recycle the umem > > > buffer back into the NIC's RX-ring or will it put it on the CQ? In > > > case of the latter, the FQ will eventually run out of buffers. > > > > It will recycle it back to the Rx ring of the NIC. The kernel will > > never take a buffer used for Rx and reuse it for Tx, or the opposite. > > A buffer put on the fill ring can only return on the Rx ring and a > > buffer put on the Tx ring can only return on the Completion ring. > > Ah! So what I wrote in my original email was incorrect - a Rx buffer > will never be put on the Completion Ring. > > I'm debugging a problem where after sending 'n' packets on the TX > ring, the app expects 'n' packets to be freed from the completion ring > - but sometimes it gets less than n (in a busy loop) or worse even > more than n packets from the completion ring - how is this possible? > Any pointers or leads are appreciated. > > This is ixgbe. Not able to reproduce on an i40e. Sounds like a bug. Maybe in the driver as you cannot reproduce it on i40e. > Please note that the app may put the same umem buffer into multiple Tx > descriptors, so the same umem buffer could be in multiple descriptors > of Tx ring and completion ring. The packet buffers are prebuilt before > transmit begins and read only during transmit, so I presume this > should not cause any problem, but would like to confirm anyway. That should be fine. > > > Is this behaviour driver specific (recycle to NIC RX-Ring or CQ) or > > > same for all drivers in ZC mode? > > > > This is core AF_XDP code, so all drivers will do the same thing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > * A Rx packet will be received in the NIC Rx Ring for Queue 0 first > > > > > > > > > * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer) > > > > > > > > > * If the program results in XDP_DROP, the driver will "free" the Umem > > > > > > > > > buffer by putting it on the Completion Ring > > > > > > > > > * If the program results in XDP_PASS, the driver will allocate a > > > > > > > > > standard Linux kernel SKB, copy the packet buffer contents into the > > > > > > > > > SKB and queue it up for standard netdev processing; It will then > > > > > > > > > "free" the RX Umem buffer by putting it on the Completion Ring (since > > > > > > > > > we have already copied packet into the skb) > > > > > > > > > > > > > > > > As the buffers are kernel buffers, user-space will not be notified. > > > > > > > > The completion ring is solely for the Tx path, saying that user space > > > > > > > > can have the buffer back. > > > > > > > > > > > > > > > > The rest is correct. > > > > > > > > > > > > > > > > /Magnus > > > > > > > > > > > > > > > > > Is this understanding correct or am I mistaken anywhere? > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > Srivats
|