(CC'ing gluster-devel) On 08/28/2012 09:38 PM, Raghavendra Gowdappa wrote:
Avati, Following are the questions/thoughts related to anonymous fd framework and their usage in quick read. Please answer or give your feedback. Questions related to anonymous fd framework: ============================================ * Anonymous fds can work because open in itself doesn't do any primary task application is interested in - like read, write etc (application does an open with an intent of doing something else). This brings in the question, why do we need open at all, can't we eliminate it altogether? If we were to eliminate open, aren't we moving from a neater to a messy design - each fop has to check whether the work associated with open (like storing contexts etc) is done in every invocation?
Some corrections to the above statement. There are two parts to the open() call 1) The effects of the call itself. Like a) Perform permission checks and establish a 'session' (with the fd) on the allowed permission [even if permission of the inode changes in the future while the fd is still open] b) Perform additional operation like file truncation when flag O_TRUNC is specified 2) Side effects of the call, like a) Specify the cache effects on future syscalls with O_[RD]SYNC, O_DIRECT flags b) Offer immunity against future calls like rename() and unlink() These are the kind of things even Gluster (or any other FS) has to guarantee with its open() syscall. Anonymous fds exist because a) Protocols like NFS3 do not support the above semantics and they are implemented completely in the client side. But we require an fd_t parameter in the read/write fops which also do not require some of the above semantics (like read/write perm checks) and other semantics are guaranteed by anonymous fds already (like immunity against rename()). Note that immunity against unlink() is currently not existing in anonymous fds. b) Internal optimizations in perf xlators do not require all the above semantics sometimes. Whether we use anonymous FDs or not, we need to keep up all the above semantics. There are some issues with the semantics even in today's version of quick-read - we assume permission check has already happened (which is usually true as FUSE performs permission checks) - but that may not be the case always. That apart, the benefit of anonymous fds in quick-read can be in handling of fd based fops in the window of time between a short-cutt'ed open() and its completion from the backend. They need not wait for the open() completion if they arrive early. Instead they can proceed with an anonymous fd -- which can significantly reduce code complexity. Again, this can be limited to O_RDONLY + ~O_DIRECT|O_TRUNC flag'ed open()s and thereby only be vulnerable to unlink()s happening in that window.
* how are ops like fsync handled with anonymous fds? How are we going to identify the fd(s) on server on which writes are actually performed? The problem is more acute if we happen to load write-behind on server side.
With the changes in http://review.gluster.com/712, an fsync() fop will be a barrier against all previous writes on the inode (no matter which fd). There is no problem if you load write-behind on the server side. fsync() is essentially an inode operation and must not discriminate writes based on the fd of origin.
* Though we are trying to decouple path from adressing an inode in glusterfs using nameless lookups, that decoupling is not complete. There are translators which use naming patterns to assign priorities to file (like io-cache, quick-read for the purposes of deciding whether to flush a cache or not). To be honest, the problem is seen only in fd-migration where we are using nameless lookups - for fresh lookups - in new graph, after a graph switch. Currently I am using nameless lookups with loc.path set, which solves the problem. Ideally nameless lookups are not the ones to be used during migration, since they are not meant to be used for fresh lookups (atleast till we get rid of dependencies on path based addressability internally in glusterfs). However, they have huge performance beniefits.
Not sure what the above point is w.r.t anonymous fds, but yes - nameless lookup takes away the sense of hierarchy (and "filename") and operations which depend on filename or hierarchy might not always work. But then this has been true even before we brought in nameless lookups as FUSE issues open() on an inode and therefore we are not guaranteed to perform open() on the right path when you have hardlinks.
Using anonymous fd framework in quick-read: =========================================== * as far as quick read goes, its task becomes very simple. Just convert the fd to anonymous during open and return. It can eliminate all the dependencies of fops having to wait till open is actually done. In fact the fops it has to implement are: lookup, open and readv.
Look at my previous comments, it must perform a little more checks. quick-read cannot just "convert" an fd to anonymous fd. Anonymous fd has fd->pid == -1 (which a quick-unwound open() fd will not). There are also other semantics which need to be met (at least with best effort) while the actual fd is still unopened.
* Anonymous fd awareness should be brought in afr. it shouldn't try to open the files in fops like writev if fd happens to be anonymous.
I think that already is the case. Also, why do you specifically mention afr? Thanks, Avati
