GlusterFS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

Our project has been through the pain of file system failures. So we
decided to implement a new FS with GlusterFS V3.03.

We has 3 servers on both east coast and west coast, and want to keep the
data synchrolized between the two location. The important files should
be replicated twice and normal files replicated once.

We designed the structure as following:

Replicate(west/east)
	Distribute West
		Replicate 1
			Server 1 Brick 1
			Server 2 Brick 1
		Replicate 2
			Server 2 Brick 2
			Server 3 Brick 2
		Replicate 3
			Server 3 Brick 3
			Server 1 Brick 3
	Distribute East
		Replicate 1
			Server 1 Brick 1
			Server 2 Brick 1
		Replicate 2
			Server 2 Brick 2
			Server 3 Brick 2
		Replicate 3
			Server 3 Brick 3
			Server 1 Brick 3

We have tested the structure on our local machines. 
It worked fine sometimes, but sometimes it had no response.

My questions are
1. Can this structure be implemented?
2. Is there a better solution for our case?
3. Listed below are the sample vol files and debug info. Are these
problems fixable?
4. Can we mount a GlusterFS, and use it as a destination folder in
Gluster server?




The vol file for servers

# Sample vol for server 1
#####################################################
volume posix1
	type storage/posix
	option directory /data1/vi1
end-volume

volume locks1
	type features/locks
	subvolumes posix1
end-volume

volume brick1
  type performance/io-threads
  option thread-count 8
  subvolumes locks1
end-volume

volume posix2
	type storage/posix
	option directory /data2/vi1
end-volume

volume locks2
	type features/locks
	subvolumes posix2
end-volume

volume brick2
  type performance/io-threads
  option thread-count 8
  subvolumes locks2
end-volume

volume posix3
	type storage/posix
	option directory /data3/vi1
end-volume

volume locks3
	type features/locks
	subvolumes posix3
end-volume

volume brick3
  type performance/io-threads
  option thread-count 8
  subvolumes locks3
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        option transport.socket.listen-port 6996
        option transport.socket.bind-address fs1
        subvolumes brick1 brick2 brick3
        option auth.addr.brick1.allow 192.168.0.*
	option auth.addr.brick2.allow 192.168.0.*
	option auth.addr.brick3.allow 192.168.0.*
end-volume


# Sample vol file for client in east coast
######################################################
  1: volume remote1-a
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host 192.168.0.195
  5:   option remote-port 6996
  6:   option remote-subvolume brick1
  7: end-volume
  8: 
  9: volume remote1-b
 10:   type protocol/client
 11:   option transport-type tcp
 12:   option remote-host 192.168.0.195
 13:   option remote-port 6997
 14:   option remote-subvolume brick1
 15: end-volume
 16: 
 17: volume remote1-c
 18:   type protocol/client
 19:   option transport-type tcp
 20:   option remote-host 192.168.0.195
 21:   option remote-port 6998
 22:   option remote-subvolume brick1
 23: end-volume
 24: 
 25: volume remote2-a
 26:   type protocol/client
 27:   option transport-type tcp
 28:   option remote-host 192.168.0.195
 29:   option remote-port 6996
 30:   option remote-subvolume brick2
 31: end-volume
 32: 
 33: volume remote2-b
 34:   type protocol/client
 35:   option transport-type tcp
 36:   option remote-host 192.168.0.195
 37:   option remote-port 6997
 38:   option remote-subvolume brick2
 39: end-volume
 40: 
 41: volume remote2-c
 42:   type protocol/client
 43:   option transport-type tcp
 44:   option remote-host 192.168.0.195
 45:   option remote-port 6998
 46:   option remote-subvolume brick2
 47: end-volume
 48: 
 49: volume remote3-a
 50:   type protocol/client
 51:   option transport-type tcp
 52:   option remote-host 192.168.0.195
 53:   option remote-port 6996
 54:   option remote-subvolume brick3
 55: end-volume
 56: 
 57: volume remote3-b
 58:   type protocol/client
 59:   option transport-type tcp
 60:   option remote-host 192.168.0.195
 61:   option remote-port 6997
 62:   option remote-subvolume brick3
 63: end-volume
 64: 
 65: volume remote3-c
 66:   type protocol/client
 67:   option transport-type tcp
 68:   option remote-host 192.168.0.195
 69:   option remote-port 6998
 70:   option remote-subvolume brick3
 71: end-volume
 72: 
 73: volume r1
 74:   type cluster/replicate
 75:   subvolumes remote1-a remote1-b
 76: end-volume
 77: 
 78: volume r2
 79:   type cluster/replicate
 80:   subvolumes remote2-b remote2-c
 81: end-volume
 82: 
 83: volume r3
 84:   type cluster/replicate
 85:   subvolumes remote3-c remote3-a
 86: end-volume
 87: 
 88: volume d-east
 89:   type cluster/distribute
 90:   subvolumes r1 r2 r3
 91: end-volume
 92: 
 93: volume w-remote1-a
 94:   type protocol/client
 95:   option transport-type tcp
 96:   option remote-host 192.168.0.194
 97:   option remote-port 6996
 98:   option remote-subvolume brick1
 99: end-volume
100: 
101: volume w-remote1-b
102:   type protocol/client
103:   option transport-type tcp
104:   option remote-host 192.168.0.194
105:   option remote-port 6997
106:   option remote-subvolume brick1
107: end-volume
108: 
109: volume w-remote1-c
110:   type protocol/client
111:   option transport-type tcp
112:   option remote-host 192.168.0.194
113:   option remote-port 6998
114:   option remote-subvolume brick1
115: end-volume
116: 
117: volume w-remote2-a
118:   type protocol/client
119:   option transport-type tcp
120:   option remote-host 192.168.0.194
121:   option remote-port 6996
122:   option remote-subvolume brick2
123: end-volume
124: 
125: volume w-remote2-b
126:   type protocol/client
127:   option transport-type tcp
128:   option remote-host 192.168.0.194
129:   option remote-port 6997
130:   option remote-subvolume brick2
131: end-volume
132: 
133: volume w-remote2-c
134:   type protocol/client
135:   option transport-type tcp
136:   option remote-host 192.168.0.194
137:   option remote-port 6998
138:   option remote-subvolume brick2
139: end-volume
140: 
141: volume w-remote3-a
142:   type protocol/client
143:   option transport-type tcp
144:   option remote-host 192.168.0.194
145:   option remote-port 6996
146:   option remote-subvolume brick3
147: end-volume
148: 
149: volume w-remote3-b
150:   type protocol/client
151:   option transport-type tcp
152:   option remote-host 192.168.0.194
153:   option remote-port 6997
154:   option remote-subvolume brick3
155: end-volume
156: 
157: volume w-remote3-c
158:   type protocol/client
159:   option transport-type tcp
160:   option remote-host 192.168.0.194
161:   option remote-port 6998
162:   option remote-subvolume brick3
163: end-volume
164: 
165: volume w-r1
166:   type cluster/replicate
167:   subvolumes w-remote1-a w-remote1-b
168: end-volume
169: 
170: volume w-r2
171:   type cluster/replicate
172:   subvolumes w-remote2-b w-remote2-c
173: end-volume
174: 
175: volume w-r3
176:   type cluster/replicate
177:   subvolumes w-remote3-c w-remote3-a
178: end-volume
179: 
180: volume d-west
181:   type cluster/distribute
182:   subvolumes w-r1 w-r2 w-r3
183: end-volume
184: 
185: volume client1
186:   type cluster/replicate
187:   option read-subvolume d-east
188:   subvolumes d-east d-west
189: end-volume
190:
191: volume trace
192:   type debug/trace
193:   option include open,close,create,readdir,opendir,closedir
194:   subvolumes client1
195: end-volume


Debug information



> +------------------------------------------------------------------------------+
> [2010-03-26 14:20:09] W [xlator.c:656:validate_xlator_volume_options] trace: option 'include' is deprecated, preferred is 'include-ops', continuing with correction
> [2010-03-26 14:20:09] C [dict.c:1142:data_to_str] dict: @data=(nil)
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] w-remote3-b: Volume is dangling. 
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] w-remote2-a: Volume is dangling. 
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] w-remote1-c: Volume is dangling. 
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] remote3-b: Volume is dangling. 
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] remote2-a: Volume is dangling. 
> [2010-03-26 14:20:09] W [client-protocol.c:6586:init] remote1-c: Volume is dangling. 
> [2010-03-26 14:20:09] N [glusterfsd.c:1396:main] glusterfs: Successfully started
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r1: Subvolume 'remote1-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r1: Subvolume 'remote1-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [fuse-bridge.c:2942:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-a' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:20:09] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:25:09] N [trace.c:1769:trace_opendir] trace: 830:( loc {path=/, ino=1}, fd=0xd1eb80)
> [2010-03-26 14:25:09] N [trace.c:808:trace_opendir_cbk] trace: 830: (op_ret=0, op_errno=117, fd=0xd1eb80)
> [2010-03-26 14:25:11] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-east (File descriptor in bad state)
> [2010-03-26 14:25:11] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-west (File descriptor in bad state)
> pending frames:
> frame : type(1) op(STATFS)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> frame : type(1) op(READDIRP)
> patchset: v3.0.2-41-g029062c
> signal received: 11
> time of crash: 2010-03-26 14:25:11
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> fdatasync 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.0.3
> /lib/libc.so.6[0x7fb9051ee040]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so[0x7fb9045860dc]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_algo_diff+0xea)[0x7fb90458675a]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_data_sync_prepare+0xdf)[0x7fb904579b6f]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_data_fix+0x1d4)[0x7fb904579e14]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0xa2)[0x7fb90457a082]
> /lib/glusterfs/3.0.3/xlator/cluster/distribute.so(dht_attr_cbk+0xd7)[0x7fb90433fdf7]
> /lib/glusterfs/3.0.3/xlator/cluster/replicate.so(afr_fstat_cbk+0xb4)[0x7fb90456c824]
> /lib/glusterfs/3.0.3/xlator/protocol/client.so(client_fstat_cbk+0x177)[0x7fb9047b1977]
> /lib/glusterfs/3.0.3/xlator/protocol/client.so(protocol_client_pollin+0xca)[0x7fb90479e1da]
> /lib/glusterfs/3.0.3/xlator/protocol/client.so(notify+0xe0)[0x7fb9047ade70]
> /lib/libglusterfs.so.0(xlator_notify+0x43)[0x7fb9059628f3]
> /lib/glusterfs/3.0.3/transport/socket.so(socket_event_handler+0xe0)[0x7fb903505720]
> /lib/libglusterfs.so.0[0x7fb90597d4cf]
> /sbin/glusterfs(main+0x90d)[0x40465d]
> /lib/libc.so.6(__libc_start_main+0xe6)[0x7fb9051d95a6]
> /sbin/glusterfs[0x402ae9]
> ---------
> ================================================================================




> [2010-03-26 14:47:32] W [xlator.c:656:validate_xlator_volume_options] trace: option 'include' is deprecated, preferred is 'include-ops', continuing with correction
> [2010-03-26 14:47:32] C [dict.c:1142:data_to_str] dict: @data=(nil)
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] w-remote3-b: Volume is dangling. 
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] w-remote2-a: Volume is dangling. 
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] w-remote1-c: Volume is dangling. 
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] remote3-b: Volume is dangling. 
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] remote2-a: Volume is dangling. 
> [2010-03-26 14:47:32] W [client-protocol.c:6586:init] remote1-c: Volume is dangling. 
> [2010-03-26 14:47:32] N [glusterfsd.c:1396:main] glusterfs: Successfully started
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r1: Subvolume 'remote1-a' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [fuse-bridge.c:2942:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote1-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-b: Connected to 192.168.0.195:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r2: Subvolume 'remote2-b' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote2-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-c: Connected to 192.168.0.195:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] r3: Subvolume 'remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] client1: Subvolume 'd-east' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] remote3-a: Connected to 192.168.0.195:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r2: Subvolume 'w-remote2-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-b' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-c: Connected to 192.168.0.194:6998, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r3: Subvolume 'w-remote3-c' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [afr.c:2627:notify] w-r1: Subvolume 'w-remote1-b' came back up; going online.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote2-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick2'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-b: Connected to 192.168.0.194:6997, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote1-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick1'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:32] N [client-protocol.c:6246:client_setvolume_cbk] w-remote3-a: Connected to 192.168.0.194:6996, attached to remote volume 'brick3'.
> [2010-03-26 14:47:45] N [trace.c:1769:trace_opendir] trace: 39:( loc {path=/, ino=1}, fd=0x22437c0)
> [2010-03-26 14:47:46] N [trace.c:808:trace_opendir_cbk] trace: 39: (op_ret=0, op_errno=117, fd=0x22437c0)
> [2010-03-26 14:47:46] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-east (File descriptor in bad state)
> [2010-03-26 14:47:46] E [afr-self-heal-algorithm.c:766:sh_diff_checksum_cbk] client1: checksum on /sample-file01 failed on subvolume d-west (File descriptor in bad state)
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote2-b: Server 192.168.0.195:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote3-c: Server 192.168.0.195:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote3-c: Server 192.168.0.195:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-b: Server 192.168.0.194:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-b: Server 192.168.0.194:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote2-c: Server 192.168.0.194:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-c: Server 192.168.0.194:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-c: Server 192.168.0.194:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote1-b: Server 192.168.0.195:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote1-a: Server 192.168.0.195:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] remote2-c: Server 192.168.0.195:6998 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:48:57] E [client-protocol.c:415:client_ping_timer_expired] w-remote2-b: Server 192.168.0.194:6997 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:10] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:10] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:10] E [client-protocol.c:415:client_ping_timer_expired] remote3-a: Server 192.168.0.195:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:39] E [client-protocol.c:415:client_ping_timer_expired] remote3-a: Server 192.168.0.195:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:39] E [client-protocol.c:415:client_ping_timer_expired] w-remote1-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.
> [2010-03-26 14:49:39] E [client-protocol.c:415:client_ping_timer_expired] w-remote3-a: Server 192.168.0.194:6996 has not responded in the last 42 seconds, disconnecting.






[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux