Hi all,
I don't think that the group intended that there be an opendirplus();
rather readdirplus() would simply be called instead of the usual
readdir(). We should clarify that.
Regarding Peter Staubach's comments about no one ever using the
readdirplus() call; well, if people weren't performing this workload in
the first place, we wouldn't *need* this sort of call! This call is
specifically targeted at improving "ls -l" performance on large
directories, and Sage has pointed out quite nicely how that might work.
In our case (PVFS), we would essentially perform three phases of
communication with the file system for a readdirplus that was obtaining
full statistics: first grabbing the directory entries, then obtaining
metadata from servers on all objects in bulk, then gathering file sizes
in bulk. The reduction in control message traffic is enormous, and the
concurrency is much greater than in a readdir()+stat()s workload. We'd
never perform this sort of optimization optimistically, as the cost of
guessing wrong is just too high. We would want to see the call as a
proper VFS operation that we could act upon.
The entire readdirplus() operation wasn't intended to be atomic, and in
fact the returned structure has space for an error associated with the
stat() on a particular entry, to allow for implementations that stat()
subsequently and get an error because the object was removed between
when the entry was read out of the directory and when the stat was
performed. I think this fits well with what Andreas and others are
thinking. We should clarify the description appropriately.
I don't think that we have a readdirpluslite() variation documented yet?
Gary? It would make a lot of sense. Except that it should probably have
a better name...
Regarding Andreas's note that he would prefer the statlite() flags to
mean "valid", that makes good sense to me (and would obviously apply to
the so-far even more hypothetical readdirpluslite()). I don't think
there's a lot of value in returning possibly-inaccurate values?
Thanks everyone,
Rob
Trond Myklebust wrote:
On Mon, 2006-12-04 at 00:32 -0700, Andreas Dilger wrote:
I'm wondering if a corresponding opendirplus() (or similar) would also be
appropriate to inform the kernel/filesystem that readdirplus() will
follow, and stat information should be gathered/buffered. Or do most
implementations wait for the first readdir() before doing any actual work
anyway?
I'm not sure what some filesystems might do here. I suppose NFS has weak
enough cache semantics that it _might_ return stale cached data from the
client in order to fill the readdirplus() data, but it is just as likely
that it ships the whole thing to the server and returns everything in
one shot. That would imply everything would be at least as up-to-date
as the opendir().
Whether or not the posix committee decides on readdirplus, I propose
that we implement this sort of thing in the kernel via a readdir
equivalent to posix_fadvise(). That can give exactly the barrier
semantics that they are asking for, and only costs 1 extra syscall as
opposed to 2 (opendirplus() and readdirplus()).
Cheers
Trond
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
