On Fri, Oct 13, 2017 at 6:03 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: > On Fri, Oct 13, 2017 at 08:50:47AM +0200, Christoph Hellwig wrote: > >> > However, chatting this over with a few more people I have an alternate >> > solution that effectively behaves the same as how non-ODP hardware >> > handles this case of hole punch / truncation today. So, today if this >> > scenario happens on a page-cache backed mapping, the file blocks are >> > unmapped and the RDMA continues into pinned pages that are no longer >> > part of the file. We can achieve the same thing with the iommu, just >> > re-target the I/O into memory that isn't part of the file. That way >> > hardware does not see I/O errors and the DAX data consistency model is >> > no worse than the page-cache case. >> >> Yikes. > > Well, as much as you say Yikes, Dan is correct, this does match the > semantics RDMA MR's already have. They become non-coherent if their > underlying object is changed, and there are many ways to get there. > I've never thought about it, but it does sound like ftruncate, > fallocate, etc on a normal file would break the MR coherency too?? > > There have been efforts in the past driven by the MPI people to > create, essentially, something like lease-break' SIGIO. Except it was > intended to be general, and wanted solve all the problems related with > MR de-coherence. This was complicated and never became acceptable to > mainline. > > Instead ODP was developed, and ODP actually solves all the problem > sanely. > > Thinking about it some more, and with your other comments on > get_user_pages in this thread, I tend to agree. It doesn't make sense > to develop a user space lease break API for MR's that is a DAX > specific feature. > > Along the some lines, it also doesn't make sense to force-invalidate > MR's linked to DAX regions, while leaving MR's linked to other > regions that have the same problem alone. > > If you want to make non-ODP MR's work better, then you need to have a > general overall solution to tell userspace when the MR becomes (or I > guess, is becoming) non-coherent, that covers all the cases that break > MR coherence, not just via DAX. > > Otherwise, I think Dan is right, keeping the current semantic of > having MRs just do something wrong, but not corrupt memory, when they > loose coherence, is broadly consistent with how non-ODP MRs work today. > I agree, keeping the current semantics is probably the best thing we could do. It's a trade-off between breaking existing applications, having a new lease API for DAX or just failing DAX in particular (as opposed to other cases). For stable mappings, what we have is probably sufficient. For mappings which could be changed, it's unclear to me how you could guarantee non-racy behavior that is bounded by a pre-defined time and guarantee no user-space errors. On top of that, ODP (should) already solve that problem transparently. IMHO, using iommu for that and causing DMA errors just because the lease broke isn't the right thing to do. > Jason Matan > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html
