On 06/01/17 11:26 AM, Jason Gunthorpe wrote: > Make a generic API for all of this and you'd have my vote.. > > IMHO, you must support basic pinning semantics - that is necessary to > support generic short lived DMA (eg filesystem, etc). That hardware > can clearly do that if it can support ODP. I agree completely. What we want is for RDMA, O_DIRECT, etc to just work with special VMAs (ie. at least those backed with ZONE_DEVICE memory). Then GPU/NVME/DAX/whatever drivers can just hand these VMAs to userspace (using whatever interface is most appropriate) and userspace can do what it pleases with them. This makes _so_ much sense and actually largely already works today (as demonstrated by iopmem). Though, of course, there are many aspects that could still be improved like denying CPU access to special VMAs and having get_user_pages avoid pinning device memory, etc, etc. But all this would just be enhancements to how VMAs work and not be effected by the basic design described above. We experimented with GPU Direct and the peer memory patchset for IB and they were broken by design. They were just a very specific hack into the IB core and thus didn't help to support O_DIRECT or any other possible DMA user. And the invalidation thing was completely nuts. We had to pray an invalidation would never occur because, if it did, our application would just break. Logan -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
