On Tue, Mar 12, 2019 at 5:17 AM Jeff Layton <jlayton@xxxxxxxxxxxxxxx> wrote: > > On Mon, 2019-03-11 at 21:59 +0000, Sage Weil wrote: > > On Mon, 11 Mar 2019, Jeff Layton wrote: > > > This may, on its face, sound like a stupid question, but I'm going over > > > the rules to ensure that our cap handling rules will be able to properly > > > support buffered creates/unlinks. > > > > > > Dentry leases seem to be really poorly documented. The "rules" for them > > > are unclear, but also the rationale. What was the supposed benefit of > > > the dentry leases in ceph over just relying on appropriate caps on the > > > parent directory? > > > > The dentry leases are granular, while the dir cap covers the whole > > directory. > > > > You get a lease on an individual dentry via a lookup, without knowing > > anything about any other sibling dentries. The direcotry might have > > thousands of files! You can satisfy a client-side lookup with a dentry > > lease. So a lookup of /home/sage/a/b/c will get a lease on /home/sage > > without having to know anything about the other zillion users, and the > > client will generally have leases on dentries connecting any active > > subtrees. > > > > OTOH, the directory Fs cap covers the entire directory. With this cap you > > can satisfy any lookup on any file in the directory if you have a positive > > hit in the client's cache. If you also have the directory's COMPLETE > > flag, you can satisfy a negative lookup locally, and also satisfy readdir > > client-side. You only get the COMPLETE flag after a full readdir without > > racing directory modifications. You could get the Fs cap on /home in teh > > example above and it would let you trust any /home/* dentries in cache, > > but you'd have to invalidate all of them on mkdir /home/newuser. > > > > The dentry leases are handed out at the MDS's discretion, so you could > > imagine the MDS deciding not to give out dentry leases on a directory it > > doesn't expect modifications on, avoiding the overhead of establishing any > > lease state. > > > > Thanks Sage. So should I take this to mean that caps on the parent > directory and dentry leases should be handled (mostly) independently? > > I ask specifically because Zheng seemed to indicate something different > with his comment here: > > http://tracker.ceph.com/issues/24461#note-16 Ah, I don't think that's what he meant. When you have Fs on the directory, you know nobody has modified its child links without your knowledge, so if you have a complete list of dentries you can satisfy negative lookups out of client cache. (And if you ALSO have an ordered list of dentries you can satisfy “ls” results, slightly contradicting what I said in my first email). When the directory’s Fs caps are revoked, you maintain leases on dentries you know about but don’t know what new ones may have been added. And then he's noting that when the client has Fsx on the directory, it doesn't have to drop them when creating a child dentry. That's because as the exclusive owner, it's in charge and knows no other client can be doing the same thing. When you only have Fs, you (like all other clients) have to drop them (or get them revoked) before a child dentry create can go through. (It's possible we could optimize this so that the creating client doesn't need to drop Fs on a create? But there are probably some subtleties around how the directory ordering works or something that make this harder than it sounds.) -Greg
