On Thu, Mar 01, 2018 at 12:27:03PM -0700, Logan Gunthorpe wrote: > > > On 01/03/18 11:42 AM, Jason Gunthorpe wrote: > >On Thu, Mar 01, 2018 at 08:35:55PM +0200, Sagi Grimberg wrote: > >This is also why I don't entirely understand why this series has a > >generic allocator for p2p mem, it makes little sense to me. > > >Why wouldn't the nmve driver just claim the entire CMB of its local > >device for its own use? Why involve p2p core code in this? > > We'd prefer to have a generic way to get p2pmem instead of restricting > ourselves to only using CMBs. We did work in the past where the P2P memory > was part of an IB adapter and not the NVMe card. So this won't work if it's > an NVMe only interface. It just seems like it it making it too complicated. In 99% of cases locality is important and you don't want to accidently use a buffer in a 3rd device that has no relation to the two doing the transfer just because it happened to be returned by the generic allocator. I can appreciate you might have some special use case for that, but it absolutely should require special configuration and not just magically happen. You bring up IB adaptor memory - if we put that into the allocator then what is to stop the NMVe driver just using it instead of the CMB buffer? That would be totally wrong in almost all cases. Seems like a very subtle and hard to debug performance trap to leave for the users, and pretty much the only reason to use P2P is performance... So why have such a dangerous interface? Jason
