On Fri, Oct 6, 2023 at 6:34 PM Magnus Karlsson <magnus.karlsson@xxxxxxxxx> wrote: > > On Fri, 6 Oct 2023 at 14:36, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > On Fri, Oct 6, 2023 at 5:23 PM Magnus Karlsson > > <magnus.karlsson@xxxxxxxxx> wrote: > > > > > > On Fri, 6 Oct 2023 at 13:41, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > > > > > On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson > > > > <magnus.karlsson@xxxxxxxxx> wrote: > > > > > > > > > > On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > > > > > > > > > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson > > > > > > <magnus.karlsson@xxxxxxxxx> wrote: > > > > > > > > > > > > > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I want to clarify my AF_XDP understanding for a particular scenario. > > > > > > > > > > > > > > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and > > > > > > > > single XDP socket bound to queue 0 with 1 Tx Ring but NO Rx Ring. > > > > > > > > Assume the NIC has only a single queue (to keep things simple for > > > > > > > > explaining this scenario). > > > > > > > > > > > > > > > > There is a XDP program attached to Queue 0 which does either a > > > > > > > > XDP_DROP or XDP_PASS for all Rx packets. > > > > > > > > > > > > > > > > We are running in Driver mode. > > > > > > > > > > > > > > > > What happens to RX packets received on Queue 0? > > > > > > > > > > > > > > > > Here's my understanding - > > > > > > > > > > > > > > > > * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring > > > > > > > > for queue 0 > > > > > > > > * The NIC Rx Ring for queue 0 is populated by the driver with UMEM > > > > > > > > buffers taken from the Fill ring > > > > > > > > > > > > > > In driver mode (i.e. not zero-copy mode) this will not happen. The > > > > > > > ring will be populated by kernel buffers. > > > > > > > > > > > > What is the behaviour in zero-copy mode? > > > > > > > > > > Then the behaviour is according to what you wrote above. Just note > > > > > that nothing will be returned in the fill ring as you are not sending > > > > > anything to the AF_XDP socket. > > > > > > > > I assume you meant "Rx Ring" and not "Fill ring" above? > > > > > > Yes, sorry. Nothing will arrive in the Rx ring, and the Fill ring will > > > not be used apart from the initial burst of buffers that will be > > > grabbed by the system. Note though that you have to provide some > > > buffers in the fill ring as these are the ones that are used by the > > > NIC in zero-copy mode. They are just never returned to user-space > > > through the Rx ring. They are just recycled in the driver so the > > > system never needs to grab more in your scenario. > > > > > > > After a XDP_DROP or XDP_PASS at RX, will the driver recycle the umem > > buffer back into the NIC's RX-ring or will it put it on the CQ? In > > case of the latter, the FQ will eventually run out of buffers. > > It will recycle it back to the Rx ring of the NIC. The kernel will > never take a buffer used for Rx and reuse it for Tx, or the opposite. > A buffer put on the fill ring can only return on the Rx ring and a > buffer put on the Tx ring can only return on the Completion ring. Ah! So what I wrote in my original email was incorrect - a Rx buffer will never be put on the Completion Ring. I'm debugging a problem where after sending 'n' packets on the TX ring, the app expects 'n' packets to be freed from the completion ring - but sometimes it gets less than n (in a busy loop) or worse even more than n packets from the completion ring - how is this possible? Any pointers or leads are appreciated. This is ixgbe. Not able to reproduce on an i40e. Please note that the app may put the same umem buffer into multiple Tx descriptors, so the same umem buffer could be in multiple descriptors of Tx ring and completion ring. The packet buffers are prebuilt before transmit begins and read only during transmit, so I presume this should not cause any problem, but would like to confirm anyway. > > Is this behaviour driver specific (recycle to NIC RX-Ring or CQ) or > > same for all drivers in ZC mode? > > This is core AF_XDP code, so all drivers will do the same thing. > > > > > > > > > > > > > > > > > > > > > > > * A Rx packet will be received in the NIC Rx Ring for Queue 0 first > > > > > > > > * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer) > > > > > > > > * If the program results in XDP_DROP, the driver will "free" the Umem > > > > > > > > buffer by putting it on the Completion Ring > > > > > > > > * If the program results in XDP_PASS, the driver will allocate a > > > > > > > > standard Linux kernel SKB, copy the packet buffer contents into the > > > > > > > > SKB and queue it up for standard netdev processing; It will then > > > > > > > > "free" the RX Umem buffer by putting it on the Completion Ring (since > > > > > > > > we have already copied packet into the skb) > > > > > > > > > > > > > > As the buffers are kernel buffers, user-space will not be notified. > > > > > > > The completion ring is solely for the Tx path, saying that user space > > > > > > > can have the buffer back. > > > > > > > > > > > > > > The rest is correct. > > > > > > > > > > > > > > /Magnus > > > > > > > > > > > > > > > Is this understanding correct or am I mistaken anywhere? > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > Srivats
|