Bugs in replication's self-heal

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



hi,

There are bugs in replication. They exist in 2.0.x, and I tried 3.0.3, it also exists, so I report the bugs here with my solution.

 
settings:  2 server(server1 and server2), 1 client with replication
 
1)
client:   [root@centos dir1]# pwd

/mnt/gluster/dir1

[root@centos dir1]# ls

ddd
 
      then kill server2;
 
client:   [root@centos dir1]# rm -rf ddd/
 
     start server2;
 
client:   [root@centos dir1]# ls
nothing. 
 
althrough the directory is empty in client. But ddd still exists in server2:

server2:[root@centos dir1]# ls

ddd

 

If client continue to do lookup, mkdir or other operations on ddd in dir1, sometimes there would be segment fault or the connection between client and server would be lost. Not every time, but do sometimes.

 

If kill server1 instead of server2, then client will recreate ddd in server2!

 
trace at client:
————————————————————————————————————————————————————
[2010-03-26 17:18:35] N [trace.c:1397:trace_rmdir] trace2: 504: (loc {path=/dir1/ddd, ino=7209308})
[2010-03-26 17:18:35] N [trace.c:833:trace_rmdir_cbk] trace2: 504: (op_ret=0, *prebuf = {st_ino=3604666, st_mode=40755, st_nlink=2, st_uid=0, st_gid=0, st_size=4096, st_blocks=16, st_atime=[Mar 26 16:16:39], st_mtime=[Mar 26 17:19:56], st_ctime=[Mar 26 17:19:56]}, *postbuf = {(null)}
[2010-03-26 17:18:35] N [trace.c:1247:trace_xattrop] trace2: 504: (path=/dir1, ino=7209322 flags=0)
[2010-03-26 17:18:35] N [trace.c:1128:trace_xattrop_cbk] trace2: 504: (op_ret=0, op_errno=0)
[2010-03-26 17:18:35] N [trace.c:1176:trace_entrylk] trace2: 504: volume=afr, (loc= {path=/dir1, ino=7209322} basename=ddd, cmd=ENTRYLK_UNLOCK, type=ENTRYLK_WRLCK)
[2010-03-26 17:18:35] N [trace.c:1113:trace_entrylk_cbk] trace2: 504: op_ret=0, op_errno=0
[2010-03-26 17:18:46] D [client-protocol.c:7041:notify] remote1: got GF_EVENT_CHILD_UP
[2010-03-26 17:18:46] D [client-protocol.c:7041:notify] remote1: got GF_EVENT_CHILD_UP
[2010-03-26 17:18:46] N [client-protocol.c:6246:client_setvolume_cbk] remote1: Connected to 192.168.0.182:7001, attached to remote volume 'trace'.
[2010-03-26 17:18:46] N [client-protocol.c:6246:client_setvolume_cbk] remote1: Connected to 192.168.0.182:7001, attached to remote volume 'trace'.
[2010-03-26 17:19:03] N [trace.c:1769:trace_opendir] trace1: 507:( loc {path=/dir1, ino=7209322}, fd=0x958b0f8)
[2010-03-26 17:19:03] N [trace.c:1769:trace_opendir] trace2: 507:( loc {path=/dir1, ino=7209322}, fd=0x958b0f8)
[2010-03-26 17:19:03] N [trace.c:808:trace_opendir_cbk] trace2: 507: (op_ret=0, op_errno=22, fd=0x958b0f8)
[2010-03-26 17:19:03] N [trace.c:808:trace_opendir_cbk] trace1: 507: (op_ret=0, op_errno=22, fd=0x958b0f8)

From trace, we can see when user was still in dir1 which the "rm" operation took place in, "ls" wouldn’t call "lookup" but call "opendir" straightly. So the changelogs store in dir1 didn’t be noticed.

 

My solution: Every time the “opendir” is called, add another “lookup” call to lookup the directory which is opendir by the “opendir”.
 
 

2)
client:    [root@centos dir2]# pwd

/mnt/gluster/dir2

[root@centos dir2]# ll ww/

-rw-r--r-- 1 root root 242 2010-03-26 file

kill server2:

client:       [root@centos dir2]# rm -rf ww

start server2:

client:       [root@centos dir2]# ll ww/

-rw-r--r-- 1 root root 242 2010-03-26 file

 

In my opinion, in the function afr_self_heal():

         if (local->success_count && local->enoent_count) {

                   afr_self_heal_missing_entries (frame, this);

         } else {

                   gf_log (this->name, GF_LOG_TRACE,

                            "proceeding to metadata check on %s",

                            local->loc.path);

                   afr_sh_missing_entries_done (frame, this);

the judge is too simple.

My solution: if (local->success_count && local->enoent_count) is ture, add some call to recursively lookup the common parent directory which exists in each server. Anyway, there must be one, for example the mount point. Lookup the first common parent directory, then let self-heal decide should whether delete or create.

 

 

Here’s my modified replicate in glusterfs-2.0.8 in the attachment:
format:
#ifndef AFR_ENTRY
    My codes
#else
    original codes
#endif
 
Besides, in the attachment, I use Hexiaobin’s code about recursive delete in Afr-self-heal-entry.c which he may report it earlier.
 

Attachment: afr.tar.gz2
Description: Binary data


[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux