On 08/30/2009 04:00 AM, Anand Avati wrote: >> I'm wondering if there's some way for glusterfs to detect the flaws of the >> underlying operating system. I believe there's no bug-free file systems in >> the universe, so I believe it is the job of the glusterfs developer to >> specify which underlying filesystem is tested and supported. It's not good >> to simply say that glusterfs works on all real-world approximations to an >> imaginary bug-free posix filesystem. >> > I would be genuinely interested to know about another project which is > geared up to be resilient against kernel hangs so that we can borrow > some ideas on how to reliably detect kernel soft lockups or syscall > hangs. As far as I know, even mature projects like Apache have not > bothered fixing such hangs (or even detecting this kind of underlying > OS flaw). > > There are projects that require kernel patches to work properly (for example, the OpenVZ project), and most Linux distributions (i.e. RedHat) maintain a set of kernel patches. Vendors may provide work arounds for known kernel problems - for example, the dovecot people go through various means to flush the NFS or FUSE cache (including for GlusterFS) before doing certain operations, and these are done using non-portable operations. Summary of it is that relying on the Linux kernel to be correct in all situations (or any kernel for that matter) will have limits. Sometimes, it is necessary to track down the problem, correct it, and provide a patch. This can involve discussions on linux-dev leading to it finally being corrected upstream, and no longer needing to provide a patch. Not saying it has to go this far - but unless the problem is understood, it shouldn't be written off either. If GlusterFS can issue a set of operations that reproducibly causes ext3 to freeze, this is of a concern for both the ext3 developers/maintainers and the GlusterFS developers/maintainers, and it is a joint problem to solve, since ext3 is so common. As for detecting lockups or hangs - I'm not aware of this being done in the userspace area, but it could be argued that this is a bit artificial of a comparison, because GlusterFS is at its base, a network file system, and it *is* common for network file systems (such as NFS) to deal with problems with the underlying volumes. GlusterFS uses FUSE as a novel approach to avoiding the problem entirely - but if GlusterFS from user space can cause the backend storage volume to freeze up, even from outside GlusterFS, then it seems like the user space barrier is insufficient. For all of the above - I am assuming that GlusterFS is being used to do something which ends up locking up the entire volume, even from outside GlusterFS. If anybody is experiencing GlusterFS *only* problems, where the underlying volume is still accessible from another process, than this would be a different problem, probably GlusterFS specific. Cheers, mark -- Mark Mielke<mark at mielke.cc>
