On Mon, Oct 9, 2017 at 12:18 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: > On Mon, Oct 09, 2017 at 12:05:30PM -0700, Dan Williams wrote: >> On Mon, Oct 9, 2017 at 11:58 AM, Jason Gunthorpe >> <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: >> > On Fri, Oct 06, 2017 at 03:35:54PM -0700, Dan Williams wrote: >> >> otherwise be quiesced. The need for this knowledge is driven by a need >> >> to make RDMA transfers to DAX mappings safe. If the DAX file's block map >> >> changes we need to be to reliably stop accesses to blocks that have been >> >> freed or re-assigned to a new file. >> > >> > If RDMA is driving this need, why not invalidate backing RDMA MRs >> > instead of requiring a IOMMU to do it? RDMA MR are finer grained and >> > do not suffer from the re-use problem David W. brought up with IOVAs.. >> >> Sounds promising. All I want in the end is to be sure that the kernel >> is enabled to stop any in-flight RDMA at will without asking >> userspace. Does this require per-RDMA driver opt-in or is there a >> common call that can be made? > > I don't think this has ever come up in the context of an all-device MR > invalidate requirement. Drivers already have code to invalidate > specifc MRs, but to find all MRs that touch certain pages and then > invalidate them would be new code. > > We also have ODP aware drivers that can retarget a MR to new > physical pages. If the block map changes DAX should synchronously > retarget the ODP MR, not halt DMA. Have a look at the patch [1], I don't touch the ODP path. > Most likely ODP & DAX would need to be used together to get robust > user applications, as having the user QP's go to an error state at > random times (due to DMA failures) during operation is never going to > be acceptable... It's not random. The process that set up the mapping and registered the memory gets SIGIO when someone else tries to modify the file map. That process then gets /proc/sys/fs/lease-break-time seconds to fix the problem before the kernel force revokes the DMA access. It's otherwise not acceptable to allow DMA into random locations when the file map changes. > Perhaps you might want to initially only support ODP MR mappings with > DAX and then the DMA fencing issue goes away? I'd rather try to fix the non-ODP DAX case instead of just turning it off. [1]: https://patchwork.kernel.org/patch/9991681/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
