Hi Greg,
I just checked the Linux client code. I think it seems more reasonable.
//code in kernel client (caps.c)
ceph_check_caps()
{
...
retain |= CEPH_CAP_ANY_SHARED;
/*
* keep RD only if we didn't have the file open RW,
* because then the mds would revoke it anyway to
* journal max_size=0.
*/
if (ci->i_max_size == 0)
retain |= CEPH_CAP_ANY_RD;
...
}
As the comment ablove, it seems only open with RW flags cause
i_max_size is non-zero. For open with RD flag, it will be zero.
So if user open a file with RD flag, kernel cilent will retain
CEPH_CAP_ANY_RD. That will decrease request number to MDS.
I'll submit a PR for fuse client as Kernel client if you think it's OK.
Marvin
On Sat, Nov 10, 2018 at 5:49 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
>
> I'm not quite sure what you're asking here. Yes, we could prioritize
> reads entirely over writes, but that means anybody opening a file for
> write would need to pay the revoke cost, not just a cap request cost.
> IIRC there's a bundle of heuristics that might lead to not dropping
> the Frc caps, but we don't want to do that unconditionally... (And
> having just the Fsr caps without Fc aren't that exciting because
> presumably you don't want to serve all IO from the OSDs, so you're
> going to be asking for Fc from the MDS anyway...)
> -Greg
>
> On Wed, Nov 7, 2018 at 10:57 PM Marvin Zhang <fanzier@xxxxxxxxx> wrote:
> >
> > Fs is cheap ,how about Fr?
> > From the client code below, after client close the file, client only
> > hold Fs and server will revoke other file cap.
> > If we change the code to let client retain both Fs and Fr, it will
> > decrease request times seding to MDS.
> > Here is what happend before changing:
> > 1. Read a files
> > 2. Client will send MetaRequest(CEPH_MDS_OP_OPEN) to MDS
> > 3. Server grant file cap Fs+Fr
> > 4. Close this file, Client::check_caps will revoke Fr.
> > 5. If you want to read again, Client::_open() will check if you have
> > Fr. If yes, don't need to send request to MDS. If no, you have to send
> > MetaRequest(CEPH_MDS_OP_OPEN) to MDS.
> > 6. If we retain Fs+Fr all the time, Client::_open() will not send
> > CEPH_MDS_OP_OPEN unless it doesnt exist in client cache.
> >
> > Client::check_caps()
> > {
> > ...
> > wanted = in->caps_wanted(); //it will be 0 after release file
> > if (wanted)
> > retain |= CEPH_CAP_ANY;
> > else
> > retain |= CEPH_CAP_ANY_SHARED; // change to retain |=
> > (CEPH_CAP_ANY_SHARED |
> > CEPH_CAP_FILE_RD)
> > ...
> > }
> > On Thu, Nov 8, 2018 at 3:11 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
> > >
> > > On Tue, Nov 6, 2018 at 8:02 PM Marvin Zhang <fanzier@xxxxxxxxx> wrote:
> > > >
> > > > Hi Greg,
> > > > I don't read server code. But from my test: I use ceph-fuse to mount
> > > > /mnt/test1 and /mnt/test2. If I open a file with Fr and don't release
> > > > the cap in /mnt/test1 . I can write the same file successfully in
> > > > /mnt/test2. So it seems client1 hold Fs and client2 can still require
> > > > Fw. Is it right?
> > >
> > > Yes, Fs is a relatively cheap capability to share out. A client
> > > holding Fs does prevent another client receiving Fx, the "exclusive"
> > > capability, on that file, though. I don't remember what all is covered
> > > by Fx, but holding it allows the client to buffer a few more kinds of
> > > updates rather than needing to go over the network and make an MDS
> > > request (with consequent Fs cap revoke rounds).
> > > -Greg
> > >
> > > >
> > > > Marvin
> > > > On Wed, Nov 7, 2018 at 3:14 AM Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
> > > > >
> > > > > On Tue, Nov 6, 2018 at 4:16 AM Jeff Layton <jlayton@xxxxxxxxxx> wrote:
> > > > > >
> > > > > > On Tue, 2018-11-06 at 15:11 +0800, Marvin Zhang wrote:
> > > > > > > Thanks for your explanation.
> > > > > > > Here is the reason why I want to clearfy the difference between Fs and Fr
> > > > > > > 1. When I open and read a file, Client::_open() will send a
> > > > > > > MetaRequest(CEPH_MDS_OP_OPEN) message to Server.
> > > > > >
> > > > > > I'm assuming you close the file after the read here?
> > > > > >
> > > > > > > 2. If I try to open this file during 5s, Client::_open() will not send
> > > > > > > MetaRequest to MDS.
> > > > > > > 3. If I try to open this file after 5s, Client::_open() will send
> > > > > > > MetaRequest to MDS, even though the file inode and dentry informations
> > > > > > > exist in client cache.
> > > > > > >
> > > > > > > And here is my question:
> > > > > > > 1. client_caps_release_delay is 5s by default. If I increase this
> > > > > > > number, client will not send MetaRequest(CEPH_MDS_OP_OPEN) too many
> > > > > > > times. Is it OK that I set this value very big, e.g 1h? Is there any
> > > > > > > bad influence?
> > > > > > > 2. in Client::check_caps()
> > > > > > > {
> > > > > > > wanted = in->caps_wanted(); //it will be 0 after release file
> > > > > > > if (wanted)
> > > > > > > retain |= CEPH_CAP_ANY;
> > > > > > > else
> > > > > > > retain |= CEPH_CAP_ANY_SHARED; //why exclude FILE_RD??
> > > > > > > //this will cause drop FILE_RD in below code, so in next
> > > > > > > Client::_open() see the cap don't have Fr, it will send a
> > > > > > > MetaRequest(CEPH_MDS_OP_OPEN) to MSD
> > > > > > > }
> > > > > > > My question is : Is is OK to set retain |= (CEPH_CAP_ANY_SHARED |
> > > > > > > CEPH_CAP_FILE_RD) ?
> > > > > > >
> > > > > >
> > > > > > That sounds reasonable at first glance. There are no real hard and fast
> > > > > > rules in the cap handling code. It really comes down to heuristics and
> > > > > > trying to minimize later cap revokes.
> > > > > >
> > > > > > Doing what you suggest shouldn't break anything, as far as I can tell.
> > > > > > It's just that in some (possibly common) situations, your MDS might end
> > > > > > up having to send more cap revokes when there is competing access. Maybe
> > > > > > it has to do that already for Fs though?
> > > > >
> > > > > Yeah, I'd be concerned less about it breaking things (though it's
> > > > > possible for the Client and MDS to get in a capabilities fight, I
> > > > > don't think this would cause it to do so) and more about what happens
> > > > > when conflicting activity is trying to occur. If a writer comes in and
> > > > > is doing active IO but this client is trying to hold Fr caps, it will
> > > > > prevent the writer from holding the "Fb" cap that lets it buffer up
> > > > > writes before dispatching them to the OSD. IIRC, holding on to the Fr
> > > > > cap will also make the MDS more likely to let the client keep the Fc
> > > > > (cache) capability, which means if a writer comes along it needs to do
> > > > > another cap revoke. Etc etc.
> > > > > -Greg
