Hello,
Our project has been through the pain of file system failures. So we
decided to implement a new FS with GlusterFS V3.03.
We has 3 servers on both east coast and west coast, and want to keep the
data synchrolized between the two location. The important files should
be replicated twice and normal files replicated once.
We designed the structure as following:
Replicate(west/east)
Distribute West
Replicate 1
Server 1 Brick 1
Server 2 Brick 1
Replicate 2
Server 2 Brick 2
Server 3 Brick 2
Replicate 3
Server 3 Brick 3
Server 1 Brick 3
Distribute East
Replicate 1
Server 1 Brick 1
Server 2 Brick 1
Replicate 2
Server 2 Brick 2
Server 3 Brick 2
Replicate 3
Server 3 Brick 3
Server 1 Brick 3
We have tested the structure on our local machines.
It worked fine sometimes, but sometimes it had no response.
My questions are
1. Can this structure be implemented?
2. Is there a better solution for our case?
3. Listed below are the sample vol files and debug info. Are these
problems fixable?
4. Can we mount a GlusterFS, and use it as a destination folder in
Gluster server?
The vol file for servers
# Sample vol for server 1
#####################################################
volume posix1
type storage/posix
option directory /data1/vi1
end-volume
volume locks1
type features/locks
subvolumes posix1
end-volume
volume brick1
type performance/io-threads
option thread-count 8
subvolumes locks1
end-volume
volume posix2
type storage/posix
option directory /data2/vi1
end-volume
volume locks2
type features/locks
subvolumes posix2
end-volume
volume brick2
type performance/io-threads
option thread-count 8
subvolumes locks2
end-volume
volume posix3
type storage/posix
option directory /data3/vi1
end-volume
volume locks3
type features/locks
subvolumes posix3
end-volume
volume brick3
type performance/io-threads
option thread-count 8
subvolumes locks3
end-volume
volume server
type protocol/server
option transport-type tcp/server
option transport.socket.listen-port 6996
option transport.socket.bind-address fs1
subvolumes brick1 brick2 brick3
option auth.addr.brick1.allow 192.168.0.*
option auth.addr.brick2.allow 192.168.0.*
option auth.addr.brick3.allow 192.168.0.*
end-volume
# Sample vol file for client in east coast
######################################################
1: volume remote1-a
2: type protocol/client
3: option transport-type tcp
4: option remote-host 192.168.0.195
5: option remote-port 6996
6: option remote-subvolume brick1
7: end-volume
8:
9: volume remote1-b
10: type protocol/client
11: option transport-type tcp
12: option remote-host 192.168.0.195
13: option remote-port 6997
14: option remote-subvolume brick1
15: end-volume
16:
17: volume remote1-c
18: type protocol/client
19: option transport-type tcp
20: option remote-host 192.168.0.195
21: option remote-port 6998
22: option remote-subvolume brick1
23: end-volume
24:
25: volume remote2-a
26: type protocol/client
27: option transport-type tcp
28: option remote-host 192.168.0.195
29: option remote-port 6996
30: option remote-subvolume brick2
31: end-volume
32:
33: volume remote2-b
34: type protocol/client
35: option transport-type tcp
36: option remote-host 192.168.0.195
37: option remote-port 6997
38: option remote-subvolume brick2
39: end-volume
40:
41: volume remote2-c
42: type protocol/client
43: option transport-type tcp
44: option remote-host 192.168.0.195
45: option remote-port 6998
46: option remote-subvolume brick2
47: end-volume
48:
49: volume remote3-a
50: type protocol/client
51: option transport-type tcp
52: option remote-host 192.168.0.195
53: option remote-port 6996
54: option remote-subvolume brick3
55: end-volume
56:
57: volume remote3-b
58: type protocol/client
59: option transport-type tcp
60: option remote-host 192.168.0.195
61: option remote-port 6997
62: option remote-subvolume brick3
63: end-volume
64:
65: volume remote3-c
66: type protocol/client
67: option transport-type tcp
68: option remote-host 192.168.0.195
69: option remote-port 6998
70: option remote-subvolume brick3
71: end-volume
72:
73: volume r1
74: type cluster/replicate
75: subvolumes remote1-a remote1-b
76: end-volume
77:
78: volume r2
79: type cluster/replicate
80: subvolumes remote2-b remote2-c
81: end-volume
82:
83: volume r3
84: type cluster/replicate
85: subvolumes remote3-c remote3-a
86: end-volume
87:
88: volume d-east
89: type cluster/distribute
90: subvolumes r1 r2 r3
91: end-volume
92:
93: volume w-remote1-a
94: type protocol/client
95: option transport-type tcp
96: option remote-host 192.168.0.194
97: option remote-port 6996
98: option remote-subvolume brick1
99: end-volume
100:
101: volume w-remote1-b
102: type protocol/client
103: option transport-type tcp
104: option remote-host 192.168.0.194
105: option remote-port 6997
106: option remote-subvolume brick1
107: end-volume
108:
109: volume w-remote1-c
110: type protocol/client
111: option transport-type tcp
112: option remote-host 192.168.0.194
113: option remote-port 6998
114: option remote-subvolume brick1
115: end-volume
116:
117: volume w-remote2-a
118: type protocol/client
119: option transport-type tcp
120: option remote-host 192.168.0.194
121: option remote-port 6996
122: option remote-subvolume brick2
123: end-volume
124:
125: volume w-remote2-b
126: type protocol/client
127: option transport-type tcp
128: option remote-host 192.168.0.194
129: option remote-port 6997
130: option remote-subvolume brick2
131: end-volume
132:
133: volume w-remote2-c
134: type protocol/client
135: option transport-type tcp
136: option remote-host 192.168.0.194
137: option remote-port 6998
138: option remote-subvolume brick2
139: end-volume
140:
141: volume w-remote3-a
142: type protocol/client
143: option transport-type tcp
144: option remote-host 192.168.0.194
145: option remote-port 6996
146: option remote-subvolume brick3
147: end-volume
148:
149: volume w-remote3-b
150: type protocol/client
151: option transport-type tcp
152: option remote-host 192.168.0.194
153: option remote-port 6997
154: option remote-subvolume brick3
155: end-volume
156:
157: volume w-remote3-c
158: type protocol/client
159: option transport-type tcp
160: option remote-host 192.168.0.194
161: option remote-port 6998
162: option remote-subvolume brick3
163: end-volume
164:
165: volume w-r1
166: type cluster/replicate
167: subvolumes w-remote1-a w-remote1-b
168: end-volume
169:
170: volume w-r2
171: type cluster/replicate
172: subvolumes w-remote2-b w-remote2-c
173: end-volume
174:
175: volume w-r3
176: type cluster/replicate
177: subvolumes w-remote3-c w-remote3-a
178: end-volume
179:
180: volume d-west
181: type cluster/distribute
182: subvolumes w-r1 w-r2 w-r3
183: end-volume
184:
185: volume client1
186: type cluster/replicate
187: option read-subvolume d-east
188: subvolumes d-east d-west
189: end-volume
190:
191: volume trace
192: type debug/trace
193: option include open,close,create,readdir,opendir,closedir
194: subvolumes client1
195: end-volume
Debug information
> +------------------------------------------------------------------------------+
> [2010-03-26 14:20:09] W [xlator.c:656:validate_xlator_volume_options] trace: option 'include' is deprecated, preferred is 'include-ops', continuing with correction
> [2010-03-26 14:20:09] C [dict.c:1142:data_to_str] dict: @data=(nil)
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] w-remote3-b: Volume is dangling.
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] w-remote2-a: Volume is dangling.
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] w-remote1-c: Volume is dangling.
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] remote3-b: Volume is dangling.
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] remote2-a: Volume is dangling.
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] remote1-c: Volume is dangling.
> [2010-03-26 14:20:09] N [glusterfsd.c:1396:main] glusterfs: Successfully started
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r1: Subvolume 'remote1-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r1: Subvolume 'remote1-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [fuse-bridge.c:2942:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:25:09] N [trace.c:1769:trace_opendir] trace: 830:( loc {path=/, ino=1}, fd=0xd1eb80)
> [2010-03-26 14:25:09] N [trace.c:808:trace_opendir_cbk] trace: 830: (op_ret=0, op_errno=117, fd=0xd1eb80)
> [2010-03-26 14:25:11] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-east (File descriptor in bad state)
> [2010-03-26 14:25:11] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-west (File descriptor in bad state)
> pending frames:
> frame : type(1) op(STATFS)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> patchset: v3.0.2-41-g029062c
> signal received: 11
> time of crash: 2010-03-26 14:25:11
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> fdatasync 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.0.3
> /lib/libc.so.6[0x7fb9051ee040]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so[0x7fb9045860dc]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_algo_diff+0xea)[0x7fb90458675a]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_data_sync_prepare+0xdf)[0x7fb904579b6f]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_data_fix+0x1d4)[0x7fb904579e14]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0xa2)[0x7fb90457a082]
> /lib/glusterfs/3.0.3/xlator/cluster/distribute.so(dht_attr_cbk+0xd7)[0x7fb90433fdf7]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_fstat_cbk+0xb4)[0x7fb90456c824]
> /lib/glusterfs/3.0.3/xlator/protocol/client.so(client_fstat_cbk+0x177)[0x7fb9047b1977]
> /lib/glusterfs/3.0.3/xlator/protocol/client.so(protocol_client_pollin+0xca)[0x7fb90479e1da]
> /lib/glusterfs/3.0.3/xlator/protocol/client.so(notify+0xe0)[0x7fb9047ade70]
> /lib/libglusterfs.so.0(xlator_notify+0x43)[0x7fb9059628f3]
> /lib/glusterfs/3.0.3/transport/socket.so(socket_event_handler+0xe0)[0x7fb903505720]
> /lib/libglusterfs.so.0[0x7fb90597d4cf]
> /sbin/glusterfs(main+0x90d)[0x40465d]
> /lib/libc.so.6(__libc_start_main+0xe6)[0x7fb9051d95a6]
> /sbin/glusterfs[0x402ae9]
> ---------
> ================================================================================
> [2010-03-26 14:47:32] W [xlator.c:656:validate_xlator_volume_options] trace: option 'include' is deprecated, preferred is 'include-ops', continuing with correction
> [2010-03-26 14:47:32] C [dict.c:1142:data_to_str] dict: @data=(nil)
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] w-remote3-b: Volume is dangling.
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] w-remote2-a: Volume is dangling.
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] w-remote1-c: Volume is dangling.
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] remote3-b: Volume is dangling.
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] remote2-a: Volume is dangling.
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] remote1-c: Volume is dangling.
> [2010-03-26 14:47:32] N [glusterfsd.c:1396:main] glusterfs: Successfully started
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r1: Subvolume 'remote1-a' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [fuse-bridge.c:2942:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-b' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-b' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:45] N [trace.c:1769:trace_opendir] trace: 39:( loc {path=/, ino=1}, fd=0x22437c0)
> [2010-03-26 14:47:46] N [trace.c:808:trace_opendir_cbk] trace: 39: (op_ret=0, op_errno=117, fd=0x22437c0)
> [2010-03-26 14:47:46] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-east (File descriptor in bad state)
> [2010-03-26 14:47:46] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-west (File descriptor in bad state)
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote2-b: Server 192.168.0.195:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote3-c: Server 192.168.0.195:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote3-c: Server 192.168.0.195:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-b: Server 192.168.0.194:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-b: Server 192.168.0.194:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote2-c: Server 192.168.0.194:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-c: Server 192.168.0.194:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-c: Server 192.168.0.194:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote1-b: Server 192.168.0.195:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote1-a: Server 192.168.0.195:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote2-c: Server 192.168.0.195:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote2-b: Server 192.168.0.194:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:10] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:10] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:10] E [client-protocol.c:415:client_ping_timer_expired] remote3-a: Server 192.168.0.195:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:39] E [client-protocol.c:415:client_ping_timer_expired] remote3-a: Server 192.168.0.195:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:39] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:39] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.
