On Wed, Feb 06, 2019 at 12:20:21PM -0800, Matthew Wilcox wrote: > On Wed, Feb 06, 2019 at 03:16:02PM -0500, Doug Ledford wrote: > > On Wed, 2019-02-06 at 11:40 -0800, Matthew Wilcox wrote: > > > On Wed, Feb 06, 2019 at 07:16:21PM +0000, Christopher Lameter wrote: > > > > > > > > though? If we only allow this use case then we may not have to worry about > > > > long term GUP because DAX mapped files will stay in the physical location > > > > regardless. > > > > > > ... except for truncate. And now that I think about it, there was a > > > desire to support hot-unplug which also needed revoke. > > > > We already support hot unplug of RDMA devices. But it is extreme. How > > does hot unplug deal with a program running from the device (something > > that would have returned ETXTBSY)? > > Not hot-unplugging the RDMA device but hot-unplugging an NV-DIMM. > > It's straightforward to migrate text pages from one DIMM to another; > you remove the PTEs from the CPU's page tables, copy the data over and > pagefaults put the new PTEs in place. We don't have a way to do similar > things to an RDMA device, do we? I've long said it is reasonable to have an emergency hard revoke for exceptional error cases - like dis-orderly hot unplug and so forth. However, IHMO a orderly migration should rely on user space to co-ordinate the migration and the application usages, including some user space driven scheme to assure forward progress.. .. and you are kind of touching on my fear here. revoke started out as only being for ftruncate. Now we need it for data migration - how soon before someone wants to do revoke just to re-balance usage/bandwidth/etc between NVDIMMS? That is now way outside of what a RDMA using system can reasonably tolerate. How would a system designer prevent this? Again, nobody is going to want a design where RDMA applications are under some undefined threat of SIGKILL - which is where any lease revoke idea is going. :( The priority of systems using RDMA is almost always to keep the RDMA working right as it is the often key service the box is providing. Jason
