On Wed, Jun 5, 2019 at 4:13 AM Jeff Layton <jlayton@xxxxxxxxxx> wrote: > > > > > I'd like to better understand what happens to file descriptors that were > > > > > opened before the blacklisting occurred. > > > > > > > > > > Suppose I hold a POSIX lock on a O_RDWR fd, and then get blacklisted. > > > > > The admin then remounts the mount. What happens to my fd? Can I carry on > > > > > using it, or will it be shut down in some fashion? What about the lock? > > > > > I've definitely lost it during the blacklisting -- how do we make the > > > > > application aware of that fact? > > > > > > > > After 'mount -f', cephfs return -EIO for any operation on open files. > > > > After remount, operations except fsync and close work as normal. fsync > > > > and close return error if any dirty data got dropped. > > > > > > > > Current cephfs code already handle file lock. after session get > > > > evicted, cephfs return -EIO for any lock operation until all local > > > > locks are released. (see commit b3f8d68f38a87) > > > > > > > > > > I think that's not sufficient. After a blacklisting event, we need to > > > shut down all file descriptors that were open before the event. > > > > > > Consider: > > > > > > Suppose I have an application (maybe a distributed database) that opens > > > a file O_RDWR and takes out a lock on the file. The client gets > > > blacklisted and loses its lock, and then the admin remounts the thing > > > leaving the fd intact. > > > > > > After the blacklisting, the application then performs some writes to the > > > file and fsyncs and that all succeeds. Those writes are now being done > > > without holding the file lock, even though the application thinks it has > > > the lock. That seems like potential silent data corruption. > > > > > > > Is it possible that application uses one file descriptor to do file > > locking, and use different file descriptors to do read/write. If it > > is, allowing read/write from new file descriptor is not safe either. > > > > Sure, applications do all sorts of crazy things. PostgreSQL (for > instance) has a process separate from the writers to handle fsyncs. That > could also be problematic in this scenario. > > The main point here is that a file description is (partly) a > representation of state granted by the MDS, and that state can no longer > be considered valid after the client has been blacklisted. > > In practice, applications would need to have special handling to detect > and deal with this situation, and >99% of them won't have it. Most > applications will need to be restarted altogether after an event like > this, which makes me question whether this is something we should do at > all. As I said in the other thread, I think one way we can address file locks is through SIGLOST (with this scenario similar to its intended use) which would be terminal for most applications. -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
