On 12/12/18 3:37 PM, Jason Gunthorpe wrote: > On Wed, Dec 12, 2018 at 04:53:49PM -0500, Jerome Glisse wrote: >>> Almost, we need some safety around assuming that DMA is complete the >>> page, so the notification would need to go all to way to userspace >>> with something like a file lease notification. It would also need to >>> be backstopped by an IOMMU in the case where the hardware does not / >>> can not stop in-flight DMA. >> >> You can always reprogram the hardware right away it will redirect >> any dma to the crappy page. > > That causes silent data corruption for RDMA users - we can't do that. > > The only way out for current hardware is to forcibly terminate the > RDMA activity somehow (and I'm not even sure this is possible, at > least it would be driver specific) > > Even the IOMMU idea probably doesn't work, I doubt all current > hardware can handle a PCI-E error TLP properly. Very true. > > On some hardware it probably just protects DAX by causing data > corruption for RDMA - I fail to see how that is a win for system > stability if the user obviously wants to use DAX and RDMA together... > > I think your approach with ODP only is the only one that meets your > requirements, the only other data-integrity-preserving approach is to > block/fail ftruncate/etc. > >> From my point of view driver should listen to ftruncate before the >> mmu notifier kicks in and send event to userspace and maybe wait >> and block ftruncate (or move it to a worker thread). > > We can do this, but we can't guarantee forward progress in userspace > and the best way we have to cancel that is portable to all RDMA > hardware is to kill the process(es).. > > So if that is acceptable then we could use user notifiers and allow > non-ODP users... > That is exactly the conclusion that some of us in the GPU world reached as well, when chatting about how this would have to work, even on modern GPU hardware that can replay page faults, in many cases. I think as long as we specify that the acceptable consequence of doing, say, umount on a filesystem that has active DMA happening is that the associated processes get killed, then we're going to be OK. What would worry me is if there was an expectation that processes could continue working properly after such a scenario. thanks, -- John Hubbard NVIDIA
