Hi, I have a small test setup on Ubuntu 12.04, using the 3.4.1-ubuntu1~precise1 packages of glusterfs from the recommended PPA. There are two gluster servers (replicate 2) and one client. Bricks are 16 GB xfs -i size=512 filesystems. All servers run on vmware. I've been using the linux kernel source for some simple performance and stability tests with many small files. When deleting the linux kernel tree with rm -Rf while rebooting one glusterfs server, it seems that some deletes are missed, or "recreated". Here's how it goes: root at client:/mnt# rm -Rf linux-3.12 At this point, I run "shutdown -r now" on one server. The deletion seems to keep running just fine, but just as the server comes back up, I get something like this on the client: rm: cannot remove `linux-3.12/arch/mips/ralink/dts': Directory not empty After the rm has run to completion: root at client:/mnt# find linux-3.12 -type f linux-3.12/arch/mips/ralink/dts/Makefile Sometimes it's more than one file, too. "gluster volume heal volname info" shows no outstanding entries. If I turn off one server before running rm, and turn it on during the rm run, a similar thing happens, only it seems worse. In one test, I had 9220 files left after rm had finished. If both servers are up during the rm run, all files are deleted as expected every time. What is happening here, and can I do something to avoid it? I was hoping that in a replica 2 cluster, you could safely reboot one server at a time (with sync-up time in between) to, say, apply OS patches without taking the gluster volume offline. I'm thankful for any help. ?ystein
