Please fine responses inline. > So now, knowing that GlusterFS is kicking into overdrive fretting about a > file it can't find, I decided to eliminate the web server altogether. I > opened up Windows Explorer, and typed in a directory that didn't exist, and > sure enough, I'm unable to navigate through the share in another Explorer > window until it finally responds again a minute later. I think the Page of > Death was exhibiting such a massive death (e.g. only able to respond again > upwards of five minutes later) because it was systematically trying to > access several files that weren't found, and each one it can't find causes > the SMB connection to hang for close to a minute. > Gluster does not really care if a file is not found. It just looks up the filename on all servers and returns -ENOENT. End of story for Gluster. What's happening here is that Samba is 'searching' through all filenames in the directory to match some other filename with strcasecmp() to provide a case-insensitive match to the user. > > I feel like this is a bit of major progress toward pinpointing the problem > for a possible resolution. Here are some additional details that may help: > > The GlusterFS directory in question, /storage, has about 80,000 subdirs in > it. As such, I'm using ext4 to overcome the subdir limitations of ext3. > The non-existent image file that is able to cause everything to freeze > exists in a directory, /storage/thisdirdoesntexist/images/blah.gif, where > "thisdirdoesntexist" is in that storage directory along with those 80,000 > real subdirs. I know it's a pretty laborious thing for Gluster to piece > together a directory listing, and combined with Joseph's recognition of the > flood of "getdents", does it seem reasonable that Gluster or Samba is > freezing because it's for some reason generating a subdir listing of > /storage whenever it can't find one of its subdirs? > Yes, it is samba searching around for the case insensitive match. As another test, if I access a file inside a non-existent subdir of a dir > that only has five subdirs, and nothing freezes. > That is because iterating over 5 names to determine non existence of a case insensitive match is trivially fast. > > So the freezing seems to be a function of the number of subdirectories that > are siblings of the first part of the path that doesn't exist, if that makes > sense. So in /this/is/a/long/path, if "is" doesn't exist, then Samba will > generate a list of subdirs under "/this". And if "/this" has 100,000 > immediate subdirs under it, then you're about to experience a world of hurt. > > I read some where that FUSE's implementation of readdir() is a blocking > operation. If true, the above explanation, plus FUSE's readdir(), are to > blame. > What do you mean by that. FUSE's readdir() is as blocking or unblocking as the rest of its open/create/getattr/setattr etc. What you probably meant was that the fuse kernel module does not cache dirents? > And I am therefore up a creek. It is not feasible to enforce the system to > only have a few subdirs at any given level to prevent the lockup. Unless > somebody, after reading this novel, has some ideas for me to try. =) Any > magical ways to not get FUSE to block, or any trickery on Samba's side? It is not FUSE blocking that is your problem. You need a quicker trick to achieve case insensitivity. Avati -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20110718/8779827f/attachment.htm>
