> -----Original Message----- > From: Jakub Kicinski [mailto:jakub.kicinski@xxxxxxxxxxxxx] > Sent: Monday, July 1, 2019 10:20 PM > To: Laatz, Kevin <kevin.laatz@xxxxxxxxx> > Cc: Jonathan Lemon <jonathan.lemon@xxxxxxxxx>; netdev@xxxxxxxxxxxxxxx; > ast@xxxxxxxxxx; daniel@xxxxxxxxxxxxx; Topel, Bjorn > <bjorn.topel@xxxxxxxxx>; Karlsson, Magnus <magnus.karlsson@xxxxxxxxx>; > bpf@xxxxxxxxxxxxxxx; intel-wired-lan@xxxxxxxxxxxxxxxx; Richardson, Bruce > <bruce.richardson@xxxxxxxxx>; Loftus, Ciara <ciara.loftus@xxxxxxxxx> > Subject: Re: [PATCH 00/11] XDP unaligned chunk placement support > > On Mon, 1 Jul 2019 15:44:29 +0100, Laatz, Kevin wrote: > > On 28/06/2019 21:29, Jonathan Lemon wrote: > > > On 28 Jun 2019, at 9:19, Laatz, Kevin wrote: > > >> On 27/06/2019 22:25, Jakub Kicinski wrote: > > >>> I think that's very limiting. What is the challenge in providing > > >>> aligned addresses, exactly? > > >> The challenges are two-fold: > > >> 1) it prevents using arbitrary buffer sizes, which will be an issue > > >> supporting e.g. jumbo frames in future. > > >> 2) higher level user-space frameworks which may want to use AF_XDP, > > >> such as DPDK, do not currently support having buffers with 'fixed' > > >> alignment. > > >> The reason that DPDK uses arbitrary placement is that: > > >> - it would stop things working on certain NICs which need > > >> the actual writable space specified in units of 1k - therefore we > > >> need 2k > > >> + metadata space. > > >> - we place padding between buffers to avoid constantly > > >> hitting the same memory channels when accessing memory. > > >> - it allows the application to choose the actual buffer > > >> size it wants to use. > > >> We make use of the above to allow us to speed up processing > > >> significantly and also reduce the packet buffer memory size. > > >> > > >> Not having arbitrary buffer alignment also means an AF_XDP > > >> driver for DPDK cannot be a drop-in replacement for existing > > >> drivers in those frameworks. Even with a new capability to allow an > > >> arbitrary buffer alignment, existing apps will need to be modified > > >> to use that new capability. > > > > > > Since all buffers in the umem are the same chunk size, the original > > > buffer address can be recalculated with some multiply/shift math. > > > However, this is more expensive than just a mask operation. > > > > Yes, we can do this. > > That'd be best, can DPDK reasonably guarantee the slicing is uniform? > E.g. it's not desperate buffer pools with different bases? It's generally uniform, but handling the crossing of (huge)page boundaries complicates things a bit. Therefore I think the final option below is best as it avoids any such problems. > > > Another option we have is to add a socket option for querying the > > metadata length from the driver (assuming it doesn't vary per packet). > > We can use that information to get back to the original address using > > subtraction. > > Unfortunately the metadata depends on the packet and how much info the > device was able to extract. So it's variable length. > > > Alternatively, we can change the Rx descriptor format to include the > > metadata length. We could do this in a couple of ways, for example, > > rather than returning the address as the start of the packet, instead > > return the buffer address that was passed in, and adding another > > 16-bit field to specify the start of packet offset with that buffer. > > If using another 16-bits of the descriptor space is not desirable, an > > alternative could be to limit umem sizes to e.g. 2^48 bits (256 > > terabytes should be enough, right :-) ) and use the remaining 16 bits > > of the address as a packet offset. Other variations on these approach > > are obviously possible too. > > Seems reasonable to me.. I think this is probably the best solution, and also has the advantage that a buffer retains its base address the full way through the cycle of Rx and Tx.
