On Thu, Oct 31, 2024 at 11:14:51AM -0400, Chuck Lever wrote: > On Wed, Oct 23, 2024 at 11:58:46AM -0400, Mike Snitzer wrote: > > We do not and cannot support file locking with NFS reexport over > > NFSv4.x for the same reason we don't do it for NFSv3: NFS reexport [ ... patch snipped ... ] > > diff --git a/Documentation/filesystems/nfs/reexport.rst b/Documentation/filesystems/nfs/reexport.rst > > index ff9ae4a46530..044be965d75e 100644 > > --- a/Documentation/filesystems/nfs/reexport.rst > > +++ b/Documentation/filesystems/nfs/reexport.rst > > @@ -26,9 +26,13 @@ Reboot recovery > > --------------- > > > > The NFS protocol's normal reboot recovery mechanisms don't work for the > > -case when the reexport server reboots. Clients will lose any locks > > -they held before the reboot, and further IO will result in errors. > > -Closing and reopening files should clear the errors. > > +case when the reexport server reboots because the source server has not > > +rebooted, and so it is not in grace. Since the source server is not in > > +grace, it cannot offer any guarantees that the file won't have been > > +changed between the locks getting lost and any attempt to recover them. > > +The same applies to delegations and any associated locks. Clients are > > +not allowed to get file locks or delegations from a reexport server, any > > +attempts will fail with operation not supported. > > > > Filehandle limits > > ----------------- Note for Mike: Last sentence "Clients are not allowed to get ... delegations from a reexport server" -- IIUC it's up to the re-export server to not hand out delegations to its clients. Still, it's important to note that NFSv4 delegation would not be available for re-exports. See below for more: I'd like this paragraph to continue to discuss the issue of OPEN and I/O behavior when the re-export server restarts. The patch seems to redact that bit of detail. Following is general discussion: > There seems to be some controversy about this approach. > > Also I think it would be nicer all around if we followed the usual > process for changes that introduce possible behavior regressions: > > - add the new behavior, make it optional, default old behavior > - wait a few releases > - change the default to new behavior > > Lastly, there haven't been any user complaints about the current > situation of no lock recovery in the re-export case. > > Jeff and I discussed this, and we plan to drop this one for 6.13 but > let the conversation continue. Mike, no action needed on your part > for the moment, but please stay tuned! > > IMO having an export option (along the lines of "async/sync") that > is documented in a man page is going to be a better plan. But if we > find a way to deal with this situation without a new administrative > control, that would be even better. Proposed solutions so far: - Disable NFS locking entirely on NFS re-export - Implement full state pass-through for re-export Some history of the NFSD design and the re-export issue is provided here: http://wiki.linux-nfs.org/wiki/index.php/NFS_re-export#reboot_recovery Certain usage scenarios require that lock state be globally visible, so disabling NFS locking on re-export mounts will need to be considered carefully. Assuming that NFSv4 LOCK operations are proliferated to the back-end server in today's NFSD, does it make sense to avoid code changes at the moment, but more carefully document the configuration options and their risks? +++ In all following configurations, no state recovery occurs when the re-export server restarts, as explained in Documentation/filesystems/nfs/reexport.rst. Mount options on the re-export server and clients: * All default: open and lock state is proliferated to the back-end server and is visible to all NFS clients. * local_lock=all on the re-export server's mount of the back-end server: clients of that server all see the same set of locks, but these locks are not visible to the back-end server or any of its clients. Open state is visible everywhere. * local_lock=all on the NFS mounts on client mounts of the re-export server: applications on NFS clients do not see locks set by applications on any other NFS clients. Open state is visible everywhere. When an NFS client of the re-export server OPENs a file, currently that creates OPEN state on the re-export server, and I assume also on the back-end server. That state cannot be recovered if the re-export server restarts, but it also cannot be blocked by a mount option. Likewise, I assume the back-end server can hand out delegations to the re-export server. If the re-export server restarts, how does it recover those delegations? The re-export server could disable delegation by blocking off its callback service, but should it? What, if anything, is being done to further develop and regularly test NFS re-export in upstream kernels? The reexport.rst file: This still reads more like design notes than administrative documentation. IMHO it should instead have a more detailed description and disclaimer regarding what kind of manual recovery is needed after a re-export server restart. That seems like important information for administrators who think they might want to deploy this solution. Maybe Documentation/ isn't the right place for administrative documentation? It might be prudent to (temporarily) label NFS re-export as experimental use only, given its incompleteness and the long list of caveats. -- Chuck Lever
