I'm not quite sure what you're asking here. Yes, we could prioritize
reads entirely over writes, but that means anybody opening a file for
write would need to pay the revoke cost, not just a cap request cost.
IIRC there's a bundle of heuristics that might lead to not dropping
the Frc caps, but we don't want to do that unconditionally... (And
having just the Fsr caps without Fc aren't that exciting because
presumably you don't want to serve all IO from the OSDs, so you're
going to be asking for Fc from the MDS anyway...)
-Greg
On Wed, Nov 7, 2018 at 10:57 PM Marvin Zhang <fanzier@xxxxxxxxx> wrote:
>
> Fs is cheap ,how about Fr?
> From the client code below, after client close the file, client only
> hold Fs and server will revoke other file cap.
> If we change the code to let client retain both Fs and Fr, it will
> decrease request times seding to MDS.
> Here is what happend before changing:
> 1. Read a files
> 2. Client will send MetaRequest(CEPH_MDS_OP_OPEN) to MDS
> 3. Server grant file cap Fs+Fr
> 4. Close this file, Client::check_caps will revoke Fr.
> 5. If you want to read again, Client::_open() will check if you have
> Fr. If yes, don't need to send request to MDS. If no, you have to send
> MetaRequest(CEPH_MDS_OP_OPEN) to MDS.
> 6. If we retain Fs+Fr all the time, Client::_open() will not send
> CEPH_MDS_OP_OPEN unless it doesnt exist in client cache.
>
> Client::check_caps()
> {
> ...
> wanted = in->caps_wanted(); //it will be 0 after release file
> if (wanted)
> retain |= CEPH_CAP_ANY;
> else
> retain |= CEPH_CAP_ANY_SHARED; // change to retain |=
> (CEPH_CAP_ANY_SHARED |
> CEPH_CAP_FILE_RD)
> ...
> }
> On Thu, Nov 8, 2018 at 3:11 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
> >
> > On Tue, Nov 6, 2018 at 8:02 PM Marvin Zhang <fanzier@xxxxxxxxx> wrote:
> > >
> > > Hi Greg,
> > > I don't read server code. But from my test: I use ceph-fuse to mount
> > > /mnt/test1 and /mnt/test2. If I open a file with Fr and don't release
> > > the cap in /mnt/test1 . I can write the same file successfully in
> > > /mnt/test2. So it seems client1 hold Fs and client2 can still require
> > > Fw. Is it right?
> >
> > Yes, Fs is a relatively cheap capability to share out. A client
> > holding Fs does prevent another client receiving Fx, the "exclusive"
> > capability, on that file, though. I don't remember what all is covered
> > by Fx, but holding it allows the client to buffer a few more kinds of
> > updates rather than needing to go over the network and make an MDS
> > request (with consequent Fs cap revoke rounds).
> > -Greg
> >
> > >
> > > Marvin
> > > On Wed, Nov 7, 2018 at 3:14 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
> > > >
> > > > On Tue, Nov 6, 2018 at 4:16 AM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
> > > > >
> > > > > On Tue, 2018-11-06 at 15:11 +0800, Marvin Zhang wrote:
> > > > > > Thanks for your explanation.
> > > > > > Here is the reason why I want to clearfy the difference between Fs and Fr
> > > > > > 1. When I open and read a file, Client::_open() will send a
> > > > > > MetaRequest(CEPH_MDS_OP_OPEN) message to Server.
> > > > >
> > > > > I'm assuming you close the file after the read here?
> > > > >
> > > > > > 2. If I try to open this file during 5s, Client::_open() will not send
> > > > > > MetaRequest to MDS.
> > > > > > 3. If I try to open this file after 5s, Client::_open() will send
> > > > > > MetaRequest to MDS, even though the file inode and dentry informations
> > > > > > exist in client cache.
> > > > > >
> > > > > > And here is my question:
> > > > > > 1. client_caps_release_delay is 5s by default. If I increase this
> > > > > > number, client will not send MetaRequest(CEPH_MDS_OP_OPEN) too many
> > > > > > times. Is it OK that I set this value very big, e.g 1h? Is there any
> > > > > > bad influence?
> > > > > > 2. in Client::check_caps()
> > > > > > {
> > > > > > wanted = in->caps_wanted(); //it will be 0 after release file
> > > > > > if (wanted)
> > > > > > retain |= CEPH_CAP_ANY;
> > > > > > else
> > > > > > retain |= CEPH_CAP_ANY_SHARED; //why exclude FILE_RD??
> > > > > > //this will cause drop FILE_RD in below code, so in next
> > > > > > Client::_open() see the cap don't have Fr, it will send a
> > > > > > MetaRequest(CEPH_MDS_OP_OPEN) to MSD
> > > > > > }
> > > > > > My question is : Is is OK to set retain |= (CEPH_CAP_ANY_SHARED |
> > > > > > CEPH_CAP_FILE_RD) ?
> > > > > >
> > > > >
> > > > > That sounds reasonable at first glance. There are no real hard and fast
> > > > > rules in the cap handling code. It really comes down to heuristics and
> > > > > trying to minimize later cap revokes.
> > > > >
> > > > > Doing what you suggest shouldn't break anything, as far as I can tell.
> > > > > It's just that in some (possibly common) situations, your MDS might end
> > > > > up having to send more cap revokes when there is competing access. Maybe
> > > > > it has to do that already for Fs though?
> > > >
> > > > Yeah, I'd be concerned less about it breaking things (though it's
> > > > possible for the Client and MDS to get in a capabilities fight, I
> > > > don't think this would cause it to do so) and more about what happens
> > > > when conflicting activity is trying to occur. If a writer comes in and
> > > > is doing active IO but this client is trying to hold Fr caps, it will
> > > > prevent the writer from holding the "Fb" cap that lets it buffer up
> > > > writes before dispatching them to the OSD. IIRC, holding on to the Fr
> > > > cap will also make the MDS more likely to let the client keep the Fc
> > > > (cache) capability, which means if a writer comes along it needs to do
> > > > another cap revoke. Etc etc.
> > > > -Greg
