On Thu, Jan 05, 2017 at 01:07:19PM -0700, Jason Gunthorpe wrote: > On Thu, Jan 05, 2017 at 02:54:24PM -0500, Jerome Glisse wrote: > > > Mellanox and NVidia support peer to peer with what they market a > > GPUDirect. It only works without IOMMU. It is probably not upstream : > > > > https://www.mail-archive.com/linux-rdma@xxxxxxxxxxxxxxx/msg21402.html > > > > I thought it was but it seems it require an out of tree driver to work. > > Right, it is out of tree and not under consideration for mainline. > > > Wether there is a vma or not isn't important to the issue anyway. If > > you want to enforce VMA rule for RDMA it is an RDMA specific discussion > > in which i don't want to be involve, it is not my turf :) > > Always having a VMA changes the discussion - the question is how to > create a VMA that reprensents IO device memory, and how do DMA > consumers extract the correct information from that VMA to pass to the > kernel DMA API so it can setup peer-peer DMA. Well my point is that it can't be. In HMM case inside a single VMA you can have one page inside GPU memory at address A but next page inside regular memory at A+4k. So handling this at the VMA level does not make sense. So in this case you would get the device from the struct page and you would query through common API to determine if you can do peer to peer. If not it would trigger migration back to regular memory. If yes then you still have to solve the IOMMU issue and hence the DMA API changes that were propose. In the GPUDirect case the idea is that you have a specific device vma that you map for peer to peer. Here thing can be at vma level and not at a page level. Expectation here is that the GPU userspace expose a special API to allow RDMA to directly happen on GPU object allocated through GPU specific API (ie it is not regular memory and it is not accessible by CPU). Both case are disjoint. Both case need to solve the IOMMU issue which seems to be best solve at the DMA API level. > > What matter is the back channel API between peer-to-peer device. Like > > the above patchset points out for GPU we need to be able to invalidate > > a mapping at any point in time. Pining is not something we want to > > live with. > > We have MMU notifiers to handle this today in RDMA. Async RDMA MR > Invalidate like you see in the above out of tree patches is totally > crazy and shouldn't be in mainline. Use ODP capable RDMA hardware. Well there is still a large base of hardware that do not have such feature and some people would like to be able to keep using those. I believe allowing direct access to GPU object that are otherwise hidden from regular kernel memory management is still meaningfull. Cheers, Jérôme -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
