Re: [PATCH 1/4] libceph: add function that reset client's entity addr

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, Jun 5, 2019 at 4:13 AM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
> > > > > I'd like to better understand what happens to file descriptors that were
> > > > > opened before the blacklisting occurred.
> > > > >
> > > > > Suppose I hold a POSIX lock on a O_RDWR fd, and then get blacklisted.
> > > > > The admin then remounts the mount. What happens to my fd? Can I carry on
> > > > > using it, or will it be shut down in some fashion? What about the lock?
> > > > > I've definitely lost it during the blacklisting -- how do we make the
> > > > > application aware of that fact?
> > > >
> > > > After 'mount -f',  cephfs return -EIO for any operation on open files.
> > > > After remount, operations except fsync and close work as normal. fsync
> > > > and close return error if any dirty data got dropped.
> > > >
> > > > Current cephfs code already handle file lock. after session get
> > > > evicted, cephfs return -EIO for any lock operation until all local
> > > > locks are released. (see commit b3f8d68f38a87)
> > > >
> > >
> > > I think that's not sufficient. After a blacklisting event, we need to
> > > shut down all file descriptors that were open before the event.
> > >
> > > Consider:
> > >
> > > Suppose I have an application (maybe a distributed database) that opens
> > > a file O_RDWR and takes out a lock on the file. The client gets
> > > blacklisted and loses its lock, and then the admin remounts the thing
> > > leaving the fd intact.
> > >
> > > After the blacklisting, the application then performs some writes to the
> > > file and fsyncs and that all succeeds. Those writes are now being done
> > > without holding the file lock, even though the application thinks it has
> > > the lock. That seems like potential silent data corruption.
> > >
> >
> > Is it possible that application uses one file descriptor to do file
> > locking, and use different file descriptors to do read/write.  If it
> > is, allowing read/write from new file descriptor is not safe either.
> >
>
> Sure, applications do all sorts of crazy things. PostgreSQL (for
> instance) has a process separate from the writers to handle fsyncs. That
> could also be problematic in this scenario.
>
> The main point here is that a file description is (partly) a
> representation of state granted by the MDS, and that state can no longer
> be considered valid after the client has been blacklisted.
>
> In practice, applications would need to have special handling to detect
> and deal with this situation, and >99% of them won't have it. Most
> applications will need to be restarted altogether after an event like
> this, which makes me question whether this is something we should do at
> all.

As I said in the other thread, I think one way we can address file
locks is through SIGLOST (with this scenario similar to its intended
use) which would be terminal for most applications.

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D



[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux