On Thu, Jan 05, 2017 at 03:42:15PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 03:19:36PM -0500, Jerome Glisse wrote: > > > > Always having a VMA changes the discussion - the question is how to > > > create a VMA that reprensents IO device memory, and how do DMA > > > consumers extract the correct information from that VMA to pass to the > > > kernel DMA API so it can setup peer-peer DMA. > > > > Well my point is that it can't be. In HMM case inside a single VMA > > you > [..] > > > In the GPUDirect case the idea is that you have a specific device vma > > that you map for peer to peer. > > [..] > > I still don't understand what you driving at - you've said in both > cases a user VMA exists. In the former case no, there is no VMA directly but if you want one than a device can provide one. But such VMA is useless as CPU access is not expected. > > From my perspective in RDMA, all I want is a core kernel flow to > convert a '__user *' into a scatter list of DMA addresses, that works no > matter what is backing that VMA, be it HMM, a 'hidden' GPU object, or > struct page memory. > > A '__user *' pointer is the only way to setup a RDMA MR, and I see no > reason to have another API at this time. > > The details of how to translate to a scatter list are a MM subject, > and the MM folks need to get > > I just don't care if that routine works at a page level, or a whole > VMA level, or some combination of both, that is up to the MM team to > figure out :) And that's what i am trying to get accross. There is 2 cases here. What exist on today hardware. Thing like GPU direct, that works on VMA level. Versus where some new hardware is going were want to do thing on page level. Both require different API at different level. What i was trying to get accross is that no matter what level you consider in the end you still need something at the DMA API level. And that the 2 different use case (device vma or regular vma) means 2 differents API for the device driver. > > > a page level. Expectation here is that the GPU userspace expose a special > > API to allow RDMA to directly happen on GPU object allocated through > > GPU specific API (ie it is not regular memory and it is not accessible > > by CPU). > > So, how do you identify these GPU objects? How do you expect RDMA > convert them to scatter lists? How will ODP work? No ODP on those. If you want vma, the GPU device driver can provide one. GPU object are disjoint from regular memory (coming from some form of mmap). They are created through ioctl and in many case are never expose to the CPU. They only exist inside the GPU driver realm. None the less there is usecase where exchanging those object accross computer over a network make sense. I am not an end user here :) > > > We have MMU notifiers to handle this today in RDMA. Async RDMA MR > > > Invalidate like you see in the above out of tree patches is totally > > > crazy and shouldn't be in mainline. Use ODP capable RDMA hardware. > > > > Well there is still a large base of hardware that do not have such > > feature and some people would like to be able to keep using those. > > Hopefully someone will figure out how to do that without the crazy > async MR invalidation. Personnaly i don't care too much about this old hardware and thus i am fine without supporting them. The open source userspace is playing catchup and doing feature for old hardware probably does not make sense. Cheers, Jérôme -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
