There was a session at the Linux Filesystem and Memory Management summit on issues that are caused by devices using get_user_pages() or elevated refcounts to pin pages and then do I/O on them. See https://lwn.net/Articles/753027/ Basically filesystems need to mark the pages readonly during writeback. Concurrent DMA into the page while it is written by a filesystem can cause corrupted data being written to the disk, cause incorrect checksums etc etc. The solution that was proposed at the meeting was that mmu notifiers can remedy that situation by allowing callbacks to the RDMA device to ensure that the RDMA device and the filesystem do not do concurrent writeback. This issue has been around for a long time and so far not caused too much grief it seems. Doing I/O to two devices from the same memory location is naturally a bit inconsistent in itself. But could we do more to prevent issues here? I think what may be useful is to not allow the memory registrations of file back writable mappings unless the device driver provides mmu callbacks or something like that. There is also the longstanding issue of the refcounts that are held over long time periods. If we require mmu notifier callbacks then we may as well go to on demand paging mode for RDMA memory registrations. This avoids increasing the refcounts long term and allows easy access control / page removal for memory management. There may even be more issues if DAX is being used but the FS writeback has the potential of biting anyone at this point it seems. I think we need to put some thought into these issues and we need some coordination between the RDMA developers and memory management. RDMA seems to be more and more important and thus its likely that issues like this will become more important.
