Lalatendu Mohanty <lmohanty at redhat.com> writes: > It sounds like a split brain issue. Below mentioned commands will help > you to figure this out. > > gluster v heal <volumeName> info split-brain > gluster v heal <volumeName> info heal-failed > > If you see any split-brain , then it is a bug. We can check with > gluster-devel if it is fixed in the master branch or there is bug for > it in bugzilla. Thank you for your reply. I've repeated a similar test on my two node cluster like this: 1. shut down node 02 2. on the client run "rm -Rf linux-3.12/" 3. while the rm is running, boot up node 02 That has the following interesting results: On the client: rm: cannot remove `linux-3.12/arch/mips/netlogic/dts': Directory not empty On the servers: "gluster v heal ovvmvol0 info split brain" and "gluster v heal ovvmvol0 info heal-failed" both show 0 entries. It also claims to have healed some files: ----- # gluster v heal ovvmvol0 info healed Gathering Heal info on volume ovvmvol0 has been successful Brick ovvm01.itea.ntnu.no:/export/sdb1/brick Number of entries: 4 at path on brick ----------------------------------- 2013-11-11 13:49:32 /linux-3.12/arch/mips/netlogic 2013-11-11 13:49:32 /linux-3.12/arch/mips 2013-11-11 13:49:30 /linux-3.12/arch 2013-11-11 13:49:29 /linux-3.12 Brick ovvm02.itea.ntnu.no:/export/sdb1/brick Number of entries: 3 at path on brick ----------------------------------- 2013-11-11 13:49:29 <gfid:2febb3e3-b72f-47f0-a6f6-cbec70d8874c>/dts/xlp_fvp.dts 2013-11-11 13:49:29 <gfid:2febb3e3-b72f-47f0-a6f6-cbec70d8874c>/dts/xlp_evp.dts 2013-11-11 13:49:29 <gfid:2febb3e3-b72f-47f0-a6f6-cbec70d8874c>/dts/Makefile ----- On the client, these three files in linux-3.12/arch/mips/netlogic/dts/ are indeed shown as present. Still curious if this was a somehow a quorum issue, I added two more servers, for a total of four servers with one brick each. Still replicate 2. I set cluster.server-quorum-type=server and cluster.server-quorum-ratio=51%. I repeated the experiment of shutting down node 02, starting an rm -Rf on a client, and booting up node 02 again. This time, it healed seemingly half the linux-3.12/arch/x86/include/asm/ directory. As one might expect, the directory is completely empty on bricks 03 and 04, while bricks 01 and 02 share the same files. Should I file a bug about this somewhere? It seems easy enough to replicate. ?ystein
