Self-heal's behavior: problem on "replace" -- it leaves garbage.

[keith at nttpc.co.jp (Keisuke TAKAHASHI)]

Hi.
I'm using GlusterFS v1.3.12 (glusterfs-1.3.12.tar.gz) via FUSE (fuse-2.7.3glfs10.tar.gz) on CentOS 5.2 x86_64 (Linux kernel 2.6.18-92.el5) now.
The nodes are HP Proliant DL360 G5 (as GlusterFS Client) and DL180 G5 (as GlusterFS Servers).
And the connections are all TCP/IP on Gigabit ethernet.

Then, I tested self-heal and I found a technical problem about "replace" -- self-heal after a node's fault and others' file-contents decreasing leaves garbage.
I would like you to show me ideas to resolve or avoid it.

First, my GlusterFS's construction is following:
  - 1 GlusterFS Client (client) and 3 GlusterFS Servers (server1,server2,server3)
  - using cluster/unify to add GlusterFS Servers
  - using cluster/afr between 3 GlsuterFS Servers underneath the cluster/unify
  - namespace volume is on the GlusterFS Client

So, self-heal will behave between server1, server2 and server3.

Now, my self-healing procedure of fault scenario is following:
  (1) Each node is active and mount point on client is /mnt/glusterfs. The operating user on client is root.
  (2) Root creates fileA and fileBC on the client local directory (not on the mount point of FUSE)
      - fileA contains strings "aaa"
      - fileBC contains strings "bbb\nccc" (\n is line break.)
  (3) Root copies fileBC on /mnt/glusterfs.
  (4) Make server2 down. (# ifdown eth0)
  (5) Root redirects fileA into fileBC (# cat fileA > fileBC)
  (6) Make server2 up. (# ifup eth0)
  (7) Now, the status of fileBC on servers is below:
      - server1: fileBC contains "aaa", trusted.glusterfs.version is 3
      - server2: fileBC contains "bbb\nccc", trusted.glusterfs.version is 2
      - server3: fileBC contains "aaa", trusted.glusterfs.version is 3
  (8) Execute self-heal. (# find /mnt/glusterfs -type f -print0 | xargs -0 head -c1 >/dev/null)
  (9) Then, the status of fileBC on servers is below:
      - server1: fileBC contains "aaa", trusted.glusterfs.version is 3
      - server2: fileBC contains "aaa\nccc", trusted.glusterfs.version is 3
      - server3: fileBC contains "aaa", trusted.glusterfs.version is 3

All right, fileBC on server2 was overwritten by others, but the result of "replace" seems in bit sequence (because original fileBC's "bbb" was replaced by "aaa" but "\nccc" was left).
In this case, the part of contents "\nccc" in fileBC on server2 looks garbage.
I would like self-heal to replace old file(s) with new file(s) completely.
Can self-heal do it? Or is there any good idea to resolve it?

Thanks.

Keisuke Takahashi


_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Keisuke TAKAHASHI / NTTPC Communications,Inc.
   E-Mail: keith at NOSPAM.nttpc.co.jp
   http://www.nttpc.co.jp/english/index.html
_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/