On Tue, Nov 6, 2018 at 8:02 PM Marvin Zhang <fanzier@xxxxxxxxx> wrote:
>
> Hi Greg,
> I don't read server code. But from my test: I use ceph-fuse to mount
> /mnt/test1 and /mnt/test2. If I open a file with Fr and don't release
> the cap in /mnt/test1 . I can write the same file successfully in
> /mnt/test2. So it seems client1 hold Fs and client2 can still require
> Fw. Is it right?
Yes, Fs is a relatively cheap capability to share out. A client
holding Fs does prevent another client receiving Fx, the "exclusive"
capability, on that file, though. I don't remember what all is covered
by Fx, but holding it allows the client to buffer a few more kinds of
updates rather than needing to go over the network and make an MDS
request (with consequent Fs cap revoke rounds).
-Greg
>
> Marvin
> On Wed, Nov 7, 2018 at 3:14 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
> >
> > On Tue, Nov 6, 2018 at 4:16 AM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
> > >
> > > On Tue, 2018-11-06 at 15:11 +0800, Marvin Zhang wrote:
> > > > Thanks for your explanation.
> > > > Here is the reason why I want to clearfy the difference between Fs and Fr
> > > > 1. When I open and read a file, Client::_open() will send a
> > > > MetaRequest(CEPH_MDS_OP_OPEN) message to Server.
> > >
> > > I'm assuming you close the file after the read here?
> > >
> > > > 2. If I try to open this file during 5s, Client::_open() will not send
> > > > MetaRequest to MDS.
> > > > 3. If I try to open this file after 5s, Client::_open() will send
> > > > MetaRequest to MDS, even though the file inode and dentry informations
> > > > exist in client cache.
> > > >
> > > > And here is my question:
> > > > 1. client_caps_release_delay is 5s by default. If I increase this
> > > > number, client will not send MetaRequest(CEPH_MDS_OP_OPEN) too many
> > > > times. Is it OK that I set this value very big, e.g 1h? Is there any
> > > > bad influence?
> > > > 2. in Client::check_caps()
> > > > {
> > > > wanted = in->caps_wanted(); //it will be 0 after release file
> > > > if (wanted)
> > > > retain |= CEPH_CAP_ANY;
> > > > else
> > > > retain |= CEPH_CAP_ANY_SHARED; //why exclude FILE_RD??
> > > > //this will cause drop FILE_RD in below code, so in next
> > > > Client::_open() see the cap don't have Fr, it will send a
> > > > MetaRequest(CEPH_MDS_OP_OPEN) to MSD
> > > > }
> > > > My question is : Is is OK to set retain |= (CEPH_CAP_ANY_SHARED |
> > > > CEPH_CAP_FILE_RD) ?
> > > >
> > >
> > > That sounds reasonable at first glance. There are no real hard and fast
> > > rules in the cap handling code. It really comes down to heuristics and
> > > trying to minimize later cap revokes.
> > >
> > > Doing what you suggest shouldn't break anything, as far as I can tell.
> > > It's just that in some (possibly common) situations, your MDS might end
> > > up having to send more cap revokes when there is competing access. Maybe
> > > it has to do that already for Fs though?
> >
> > Yeah, I'd be concerned less about it breaking things (though it's
> > possible for the Client and MDS to get in a capabilities fight, I
> > don't think this would cause it to do so) and more about what happens
> > when conflicting activity is trying to occur. If a writer comes in and
> > is doing active IO but this client is trying to hold Fr caps, it will
> > prevent the writer from holding the "Fb" cap that lets it buffer up
> > writes before dispatching them to the OSD. IIRC, holding on to the Fr
> > cap will also make the MDS more likely to let the client keep the Fc
> > (cache) capability, which means if a writer comes along it needs to do
> > another cap revoke. Etc etc.
> > -Greg
