Hello Gluster gurus, I'm experimenting with Gluster and I've run across a problem that I don't understand. I have Ubuntu 12.04 with Gluster 3.2.5. I created a volume and filled it with data. Then I left an rm -fr going to wipe it out over night. When I came in this morning, I found a lot of stuff like this: rm: cannot remove `old/old.5/test-host/20130707/etc/init.d/apport': Stale NFS file handle The log file data-testvol.log shows: [2013-07-09 10:13:37.229796] W [stat-prefetch.c:2632:sp_unlink_helper] 0-testvol-stat-prefetch: lookup-behind has failed for path (/old/old.5/test-host/20130707/etc/init.d/apport)(Stale NFS file handle), unwinding unlink call waiting on it [2013-07-09 10:13:37.229811] W [fuse-bridge.c:911:fuse_unlink_cbk] 0-glusterfs-fuse: 105965: UNLINK() /old/old.5/test-host/20130707/etc/init.d/apport => -1 (Stale NFS file handle) The interesting thing is that if I delete old/old.5/test-host/20130707/etc/init.d/apport by hand, it works: root at gluster-server-a:/data/testvol# cd old/old.5/test-host/20130707/etc/init.d root at gluster-server-a:/data/testvol/old/old.5/test-host/20130707/etc/init.d# rm apport This seems to have worked for files, and for some directories. But for other directories there's still a problem. Gluster seems to think that some directories are not empty, even though there is nothing inside: root at gluster-server-a:/data/testvol/old/old.5/[...stuff...]/Maildir# ls -al total 72 drwxr-xrwx 3 root root 16384 Jul 8 23:46 . drwxr-xrwx 3 root root 16384 Jul 9 00:10 .. drwxr-xr-x 2 root root 16384 Jul 9 10:27 .Old Mail.Mailbox12092004_end root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir# cd .Old\ Mail.Mailbox12092004_end/ root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir/.Old Mail.Mailbox12092004_end# ls -al total 48 drwxr-xr-x 3 root root 16384 Jul 9 10:27 . drwxr-xrwx 3 root root 16384 Jul 8 23:46 .. root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir/.Old Mail.Mailbox12092004_end# cd .. root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir# rmdir .Old\ Mail.Mailbox12092004_end/ rmdir: failed to remove `.Old Mail.Mailbox12092004_end/': Directory not empty root at gluster-server-a:/data/bkupc1/old/old.5/[...stuff...]/Maildir# rm -fr .Old\ Mail.Mailbox12092004_end/ rm: cannot remove `.Old Mail.Mailbox12092004_end': Directory not empty Next I rebooted every node in the cluster. The problem persists. Anyone have an idea what's going on? Or how to fix it? Michael
