On Dec 05, 2006 10:23 -0500, Trond Myklebust wrote: > On Tue, 2006-12-05 at 03:26 -0700, Andreas Dilger wrote: > > I think the "barrier semantics" are something that have just crept > > into this discussion and is confusing the issue. > > It is the _only_ concept that is of interest for something like NFS or > CIFS. We already have the ability to cache the information. Actually, wouldn't the ability for readdirplus() (with valid flag) be useful for NFS if only to indicate that it does not need to flush the cache because the ctime/mtime isn't needed by the caller? > 'find' should be quite happy with the existing readdir(). It does not > need to use stat() or readdirplus() in order to recurse because > readdir() provides d_type. It does in any but the most simplistic invocations, like "find -mtime" or "find -mode" or "find -uid", etc. > If the application is able to select a statlite()-type of behaviour with > the fadvise() hints, your filesystem could be told to serve up cached > information instead of regrabbing locks. In fact that is a much more > flexible scheme, since it also allows the filesystem to background the > actual inode lookups, or to defer them altogether if that is more > efficient. I guess I just don't understand how fadvise() on a directory file handle (used for readdir()) can be used to affect later stat operations (which definitely will NOT be using that file handle)? If you mean that the application should actually open() each file, fadvise(), fstat(), close(), instead of just a stat() call then we are WAY into negative improvements here due to overhead of doing open+close. > > The filesystem can't always do "stat-ahead" on the files because that > > requires instantiating an inode on the client which may be stale (lock > > revoked) by the time the app gets to it, and the app (and the VFS) have > > no idea just how stale it is, and whether the stat is a "real" stat or > > "only" the readdir stat (because the fadvise would only be useful on > > the directory, and not all of the child entries), so it would need to > > re-stat the file. > > Then provide hints that allow the app to select which behaviour it > prefers. Most (all?) apps don't _care_, and so would be quite happy with > cached information. That is why the current NFS caching model exists in > the first place. Most clustered filesystems have strong cache semantics, so that isn't a problem. IMHO, the mechanism to pass the hint to the filesystem IS the readdirplus_lite() that tells the filesystem exactly which data is needed on each directory entry. > > Also, this would potentially blow the client's real > > working set of inodes out of cache. > > Why? Because in many cases it is desirable to limit the number of DLM locks on a given client (e.g. GFS2 thread with AKPM about clients with millions of DLM locks due to lack of memory pressure on large mem systems). That means a finite-size lock LRU on the client that risks being wiped out by a few thousand files in a directory doing "readdir() + 5000*stat()". Consider a system like BlueGene/L with 128k compute cores. Jobs that run on that system will periodically (e.g. every hour) create up to 128K checkpoint+restart files to avoid losing a lot of computation if a node crashes. Even if each one of the checkpoints is in a separate directory (I wish all users were so nice :-) it means 128K inodes+DLM locks for doing an "ls" in the directory. > > Doing things en-masse with readdirplus() also allows the filesystem to > > do the stat() operations in parallel internally (which is a net win if > > there are many servers involved) instead of serially as the application > > would do. > > If your application really cared, it could add threading to 'ls' to > achieve the same result. You can also have the filesystem preload that > information based on fadvise hints. But it would still need 128K RPCs to get that information, and 128K new inodes on that client. And what is the chance that I can get a multi-threading "ls" into the upstream GNU ls code? In the case of local filesystems multi-threading ls would be a net loss due to seeking. But even for local filesystems readdirplus_lite() would allow them to fill in stat information they already have (either in cache or on disk), and may avoid doing extra work that isn't needed. For filesystems that don't care, readdirplus_lite() can just be readdir()+stat() internally. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
