I'm running glusterfs 3.1.1, 4 volumes distributed, tcp transport, fuse clients. The servers are running CentOS 5.5, the clients CentOS 5.4. The symptom I'm having which reproduces the most frequently goes like this. There are a bunch of jobs (several hundred) running on various client nodes (~50), but for no obvious reason one or several of them will get a failure to open a file (errno 13, permission denied). Subsequently it will not be possible to delete that directory because it is not empty. Attempts to delete the files which make it non-empty fail, "no such file or directory". I attached the program and driver script which I use to reproduce this, but not some log files (I ran them with trace and even compressed they are 150M). It is not perfect, it only reproduces maybe one-third of the runs. Here's a typical output: $ ATTACK-QRSH Quantity of jobs: 1000 Mon Dec 20 14:59:52 EST 2010: rm ..... Mon Dec 20 14:59:52 EST 2010: create data ..... Mon Dec 20 14:59:54 EST 2010: wait for creates..... Couldn't create data/051/307-this!"":is:!""the""filename""and""it""is""very""strange.tmp: errno 13 (Permission denied) Couldn't create data/051/563-this!"":is:!""the""filename""and""it""is""very""strange.tmp: errno 13 (Permission denied) Mon Dec 20 15:04:45 EST 2010: check data ..... Mon Dec 20 15:04:47 EST 2010: wait for checks..... Unable to reopen for read data/051/563-this!"":is:!""the""filename""and""it""is""very""strange, errno 2 (No such file or directory) Unable to reopen for read data/051/307-this!"":is:!""the""filename""and""it""is""very""strange, errno 2 (No such file or directory) Mon Dec 20 15:08:53 EST 2010: rm again ..... rm: cannot remove directory `data/051': Directory not empty Mon Dec 20 15:09:16 EST 2010: done $ cd data/051 $ ls 179-this!"":is:!""the""filename""and""it""is""very""strange 435-this!"":is:!""the""filename""and""it""is""very""strange $ rm * rm: cannot remove `179-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory rm: cannot remove `435-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory $ cd .. $ rm -r 051 rm: cannot remove `051/179-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory rm: cannot remove `051/435-this!"":is:!""the""filename""and""it""is""very""strange': No such file or directory Note that the files which won't delete are in the same directory as had the permission error earlier. .. Lana (lana.deere at gmail.com)
