mount
[2014-10-13 17:36:56.758654] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs --direct-io-mode=enable --fuse-mountopts=default_permissions,allow_other,max_read=131072 --volfile-server=stor1 --volfile-server=stor2 --volfile-id=HA-WIN-TT-1T --fuse-mountopts=default_permissions,allow_other,max_read=131072 /srv/nfs/HA-WIN-TT-1T)
[2014-10-13 17:36:56.762162] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-10-13 17:36:56.762223] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-10-13 17:36:56.766686] I [dht-shared.c:311:dht_init_regex] 0-HA-WIN-TT-1T-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2014-10-13 17:36:56.768887] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-1: SSL support is NOT enabled
[2014-10-13 17:36:56.768939] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-1: using system polling thread
[2014-10-13 17:36:56.769280] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-0: SSL support is NOT enabled
[2014-10-13 17:36:56.769294] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-0: using system polling thread
[2014-10-13 17:36:56.769336] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-0: parent translators are ready, attempting connect on transport
[2014-10-13 17:36:56.769829] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-1: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume HA-WIN-TT-1T-client-0
2: type protocol/client
3: option remote-host stor1
4: option remote-subvolume /exports/NFS-WIN/1T
5: option transport-type socket
6: option ping-timeout 10
7: option send-gids true
8: end-volume
9:
10: volume HA-WIN-TT-1T-client-1
11: type protocol/client
12: option remote-host stor2
13: option remote-subvolume /exports/NFS-WIN/1T
14: option transport-type socket
15: option ping-timeout 10
16: option send-gids true
17: end-volume
18:
19: volume HA-WIN-TT-1T-replicate-0
20: type cluster/replicate
21: subvolumes HA-WIN-TT-1T-client-0 HA-WIN-TT-1T-client-1
22: end-volume
23:
24: volume HA-WIN-TT-1T-dht
25: type cluster/distribute
26: subvolumes HA-WIN-TT-1T-replicate-0
27: end-volume
28:
29: volume HA-WIN-TT-1T-write-behind
30: type performance/write-behind
31: subvolumes HA-WIN-TT-1T-dht
32: end-volume
33:
34: volume HA-WIN-TT-1T-read-ahead
35: type performance/read-ahead
36: subvolumes HA-WIN-TT-1T-write-behind
37: end-volume
38:
39: volume HA-WIN-TT-1T-io-cache
40: type performance/io-cache
41: subvolumes HA-WIN-TT-1T-read-ahead
42: end-volume
43:
44: volume HA-WIN-TT-1T-quick-read
45: type performance/quick-read
46: subvolumes HA-WIN-TT-1T-io-cache
47: end-volume
48:
49: volume HA-WIN-TT-1T-open-behind
50: type performance/open-behind
51: subvolumes HA-WIN-TT-1T-quick-read
52: end-volume
53:
54: volume HA-WIN-TT-1T-md-cache
55: type performance/md-cache
56: subvolumes HA-WIN-TT-1T-open-behind
57: end-volume
58:
59: volume HA-WIN-TT-1T
60: type debug/io-stats
61: option latency-measurement off
62: option count-fop-hits off
63: subvolumes HA-WIN-TT-1T-md-cache
64: end-volume
65:
+------------------------------------------------------------------------------+
[2014-10-13 17:36:56.770718] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-1: changing port to 49160 (from 0)
[2014-10-13 17:36:56.771378] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-0: changing port to 49160 (from 0)
[2014-10-13 17:36:56.772008] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:36:56.772083] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:36:56.772338] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Connected to
10.250.0.2:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:36:56.772361] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:36:56.772424] I [afr-common.c:4131:afr_notify] 0-HA-WIN-TT-1T-replicate-0: Subvolume 'HA-WIN-TT-1T-client-1' came back up; going online.
[2014-10-13 17:36:56.772463] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Connected to
10.250.0.1:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:36:56.772477] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:36:56.779099] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 0
[2014-10-13 17:36:56.779338] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-0: Server lk version = 1
[2014-10-13 17:36:56.779367] I [fuse-bridge.c:3914:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.17
[2014-10-13 17:36:56.779438] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-1: Server lk version = 1
[2014-10-13 17:37:02.010942] I [fuse-bridge.c:4818:fuse_thread_proc] 0-fuse: unmounting /srv/nfs/HA-WIN-TT-1T
[2014-10-13 17:37:02.011296] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fc7b7672e6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7fc7b7d20b50] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fc7b95add55]))) 0-: received signum (15), shutting down
[2014-10-13 17:37:02.011316] I [fuse-bridge.c:5475:fini] 0-fuse: Unmounting '/srv/nfs/HA-WIN-TT-1T'.
[2014-10-13 17:37:31.133036] W [socket.c:522:__socket_rwv] 0-HA-WIN-TT-1T-client-0: readv on
10.250.0.1:49160 failed (No data available)
[2014-10-13 17:37:31.133110] I [client.c:2229:client_rpc_notify] 0-HA-WIN-TT-1T-client-0: disconnected from
10.250.0.1:49160. Client process will keep trying to connect to glusterd until brick's port is available
[2014-10-13 17:37:33.317437] W [socket.c:522:__socket_rwv] 0-HA-WIN-TT-1T-client-1: readv on
10.250.0.2:49160 failed (No data available)
[2014-10-13 17:37:33.317478] I [client.c:2229:client_rpc_notify] 0-HA-WIN-TT-1T-client-1: disconnected from
10.250.0.2:49160. Client process will keep trying to connect to glusterd until brick's port is available
[2014-10-13 17:37:33.317496] E [afr-common.c:4168:afr_notify] 0-HA-WIN-TT-1T-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2014-10-13 17:37:42.045604] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-0: changing port to 49160 (from 0)
[2014-10-13 17:37:42.046177] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:37:42.048863] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Connected to
10.250.0.1:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:37:42.048883] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:37:42.048897] I [client-handshake.c:1314:client_post_handshake] 0-HA-WIN-TT-1T-client-0: 1 fds open - Delaying child_up until they are re-opened
[2014-10-13 17:37:42.049299] W [client-handshake.c:980:client3_3_reopen_cbk] 0-HA-WIN-TT-1T-client-0: reopen on <gfid:b00e322a-7bae-479f-91e0-1fd77c73692b> failed (Stale NFS file handle)
[2014-10-13 17:37:42.049328] I [client-handshake.c:936:client_child_up_reopen_done] 0-HA-WIN-TT-1T-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2014-10-13 17:37:42.049360] I [afr-common.c:4131:afr_notify] 0-HA-WIN-TT-1T-replicate-0: Subvolume 'HA-WIN-TT-1T-client-0' came back up; going online.
[2014-10-13 17:37:42.049446] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-0: Server lk version = 1
[2014-10-13 17:37:45.087592] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-1: changing port to 49160 (from 0)
[2014-10-13 17:37:45.088132] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:37:45.088343] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Connected to
10.250.0.2:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:37:45.088360] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:37:45.088373] I [client-handshake.c:1314:client_post_handshake] 0-HA-WIN-TT-1T-client-1: 1 fds open - Delaying child_up until they are re-opened
[2014-10-13 17:37:45.088681] W [client-handshake.c:980:client3_3_reopen_cbk] 0-HA-WIN-TT-1T-client-1: reopen on <gfid:b00e322a-7bae-479f-91e0-1fd77c73692b> failed (Stale NFS file handle)
[2014-10-13 17:37:45.088697] I [client-handshake.c:936:client_child_up_reopen_done] 0-HA-WIN-TT-1T-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2014-10-13 17:37:45.088819] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-1: Server lk version = 1
[2014-10-13 17:37:54.601822] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs --direct-io-mode=enable --fuse-mountopts=default_permissions,allow_other,max_read=131072 --volfile-server=stor1 --volfile-server=stor2 --volfile-id=HA-WIN-TT-1T --fuse-mountopts=default_permissions,allow_other,max_read=131072 /srv/nfs/HA-WIN-TT-1T)
[2014-10-13 17:37:54.604972] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-10-13 17:37:54.605034] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-10-13 17:37:54.609219] I [dht-shared.c:311:dht_init_regex] 0-HA-WIN-TT-1T-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2014-10-13 17:37:54.611421] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-1: SSL support is NOT enabled
[2014-10-13 17:37:54.611466] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-1: using system polling thread
[2014-10-13 17:37:54.611808] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-0: SSL support is NOT enabled
[2014-10-13 17:37:54.611821] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-0: using system polling thread
[2014-10-13 17:37:54.611862] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-0: parent translators are ready, attempting connect on transport
[2014-10-13 17:37:54.612354] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-1: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume HA-WIN-TT-1T-client-0
2: type protocol/client
3: option remote-host stor1
4: option remote-subvolume /exports/NFS-WIN/1T
5: option transport-type socket
6: option ping-timeout 10
7: option send-gids true
8: end-volume
9:
10: volume HA-WIN-TT-1T-client-1
11: type protocol/client
12: option remote-host stor2
13: option remote-subvolume /exports/NFS-WIN/1T
14: option transport-type socket
15: option ping-timeout 10
16: option send-gids true
17: end-volume
18:
19: volume HA-WIN-TT-1T-replicate-0
20: type cluster/replicate
21: subvolumes HA-WIN-TT-1T-client-0 HA-WIN-TT-1T-client-1
22: end-volume
23:
24: volume HA-WIN-TT-1T-dht
25: type cluster/distribute
26: subvolumes HA-WIN-TT-1T-replicate-0
27: end-volume
28:
29: volume HA-WIN-TT-1T-write-behind
30: type performance/write-behind
31: subvolumes HA-WIN-TT-1T-dht
32: end-volume
33:
34: volume HA-WIN-TT-1T-read-ahead
35: type performance/read-ahead
36: subvolumes HA-WIN-TT-1T-write-behind
37: end-volume
38:
39: volume HA-WIN-TT-1T-io-cache
40: type performance/io-cache
41: subvolumes HA-WIN-TT-1T-read-ahead
42: end-volume
43:
44: volume HA-WIN-TT-1T-quick-read
45: type performance/quick-read
46: subvolumes HA-WIN-TT-1T-io-cache
47: end-volume
48:
49: volume HA-WIN-TT-1T-open-behind
50: type performance/open-behind
51: subvolumes HA-WIN-TT-1T-quick-read
52: end-volume
53:
54: volume HA-WIN-TT-1T-md-cache
55: type performance/md-cache
56: subvolumes HA-WIN-TT-1T-open-behind
57: end-volume
58:
59: volume HA-WIN-TT-1T
60: type debug/io-stats
61: option latency-measurement off
62: option count-fop-hits off
63: subvolumes HA-WIN-TT-1T-md-cache
64: end-volume
65:
+------------------------------------------------------------------------------+
[2014-10-13 17:37:54.613137] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-0: changing port to 49160 (from 0)
[2014-10-13 17:37:54.613521] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-1: changing port to 49160 (from 0)
[2014-10-13 17:37:54.614228] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:37:54.614399] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:37:54.614483] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Connected to
10.250.0.1:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:37:54.614499] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:37:54.614557] I [afr-common.c:4131:afr_notify] 0-HA-WIN-TT-1T-replicate-0: Subvolume 'HA-WIN-TT-1T-client-0' came back up; going online.
[2014-10-13 17:37:54.614625] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-0: Server lk version = 1
[2014-10-13 17:37:54.614709] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Connected to
10.250.0.2:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:37:54.614724] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:37:54.621318] I [fuse-bridge.c:4977:fuse_graph_setup] 0-fuse: switched to graph 0
[2014-10-13 17:37:54.621545] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-1: Server lk version = 1
[2014-10-13 17:37:54.621617] I [fuse-bridge.c:3914:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.17
[2014-10-13 17:38:25.951778] W [client-rpc-fops.c:4235:client3_3_flush] 0-HA-WIN-TT-1T-client-0: (b00e322a-7bae-479f-91e0-1fd77c73692b) remote_fd is -1. EBADFD
[2014-10-13 17:38:25.951827] W [client-rpc-fops.c:4235:client3_3_flush] 0-HA-WIN-TT-1T-client-1: (b00e322a-7bae-479f-91e0-1fd77c73692b) remote_fd is -1. EBADFD
[2014-10-13 17:38:25.966963] I [fuse-bridge.c:4818:fuse_thread_proc] 0-fuse: unmounting /srv/nfs/HA-WIN-TT-1T
[2014-10-13 17:38:25.967174] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ffec893de6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7ffec8febb50] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7ffeca878d55]))) 0-: received signum (15), shutting down
[2014-10-13 17:38:25.967194] I [fuse-bridge.c:5475:fini] 0-fuse: Unmounting '/srv/nfs/HA-WIN-TT-1T'.
[2014-10-13 17:40:21.500514] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:40:21.517782] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:40:21.524056] I [dht-shared.c:311:dht_init_regex] 0-HA-WIN-TT-1T-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2014-10-13 17:40:21.528430] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
glusterfshd stor1
2014-10-13 17:38:17.203360] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/75bbc77a676bde0d0afe20f40dc9e3e1.socket --xlator-option *replicate*.node-uuid=e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3)
[2014-10-13 17:38:17.204958] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
[2014-10-13 17:38:17.205016] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
[2014-10-13 17:38:17.205188] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-10-13 17:38:17.205209] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-10-13 17:38:17.207840] I [graph.c:254:gf_add_cmdline_options] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: adding option 'node-uuid' for volume 'HA-2TB-TT-Proxmox-cluster-replicate-0' with value 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
[2014-10-13 17:38:17.209433] I [socket.c:3561:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-1: SSL support is NOT enabled
[2014-10-13 17:38:17.209448] I [socket.c:3576:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-1: using system polling thread
[2014-10-13 17:38:17.209625] I [socket.c:3561:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-0: SSL support is NOT enabled
[2014-10-13 17:38:17.209634] I [socket.c:3576:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-0: using system polling thread
[2014-10-13 17:38:17.209652] I [client.c:2294:notify] 0-HA-2TB-TT-Proxmox-cluster-client-0: parent translators are ready, attempting connect on transport
[2014-10-13 17:38:17.210241] I [client.c:2294:notify] 0-HA-2TB-TT-Proxmox-cluster-client-1: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume HA-2TB-TT-Proxmox-cluster-client-0
2: type protocol/client
3: option remote-host stor1
4: option remote-subvolume /exports/HA-2TB-TT-Proxmox-cluster/2TB
5: option transport-type socket
6: option username 59c66122-55c1-4c28-956e-6189fcb1aff5
7: option password 34b79afb-a93c-431b-900a-b688e67cdbc9
8: option ping-timeout 10
9: end-volume
10:
11: volume HA-2TB-TT-Proxmox-cluster-client-1
12: type protocol/client
13: option remote-host stor2
14: option remote-subvolume /exports/HA-2TB-TT-Proxmox-cluster/2TB
15: option transport-type socket
16: option username 59c66122-55c1-4c28-956e-6189fcb1aff5
17: option password 34b79afb-a93c-431b-900a-b688e67cdbc9
18: option ping-timeout 10
19: end-volume
20:
21: volume HA-2TB-TT-Proxmox-cluster-replicate-0
22: type cluster/replicate
23: option node-uuid e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
24: option background-self-heal-count 0
25: option metadata-self-heal on
26: option data-self-heal on
27: option entry-self-heal on
28: option self-heal-daemon on
29: option iam-self-heal-daemon yes
30: subvolumes HA-2TB-TT-Proxmox-cluster-client-0 HA-2TB-TT-Proxmox-cluster-client-1
31: end-volume
32:
33: volume glustershd
34: type debug/io-stats
35: subvolumes HA-2TB-TT-Proxmox-cluster-replicate-0
36: end-volume
37:
+------------------------------------------------------------------------------+
[2014-10-13 17:38:17.210709] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-2TB-TT-Proxmox-cluster-client-0: changing port to 49159 (from 0)
[2014-10-13 17:38:17.211008] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-2TB-TT-Proxmox-cluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:17.211170] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Connected to
10.250.0.1:49159, attached to remote volume '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
[2014-10-13 17:38:17.211195] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:17.211250] I [afr-common.c:4131:afr_notify] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Subvolume 'HA-2TB-TT-Proxmox-cluster-client-0' came back up; going online.
[2014-10-13 17:38:17.211297] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Server lk version = 1
[2014-10-13 17:38:17.211656] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Another crawl is in progress for HA-2TB-TT-Proxmox-cluster-client-0
[2014-10-13 17:38:17.211661] E [afr-self-heald.c:1479:afr_find_child_position] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: getxattr failed on HA-2TB-TT-Proxmox-cluster-client-1 - (Transport endpoint is not connected)
[2014-10-13 17:38:17.216327] E [afr-self-heal-data.c:1611:afr_sh_data_open_cbk] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: open of <gfid:65381af4-8e0b-4721-8214-71d29dcf5237> failed on child HA-2TB-TT-Proxmox-cluster-client-1 (Transport endpoint is not connected)
[2014-10-13 17:38:17.217372] E [afr-self-heal-data.c:1611:afr_sh_data_open_cbk] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: open of <gfid:65381af4-8e0b-4721-8214-71d29dcf5237> failed on child HA-2TB-TT-Proxmox-cluster-client-1 (Transport endpoint is not connected)
[2014-10-13 17:38:19.226057] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-2TB-TT-Proxmox-cluster-client-1: changing port to 49159 (from 0)
[2014-10-13 17:38:19.226704] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-2TB-TT-Proxmox-cluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:19.226896] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Connected to
10.250.0.2:49159, attached to remote volume '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
[2014-10-13 17:38:19.226916] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:19.227031] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Server lk version = 1
[2014-10-13 17:38:25.933950] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1a7c03ce6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1a7c6eab50] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7f1a7df77d55]))) 0-: received signum (15), shutting down
[2014-10-13 17:38:26.942918] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/75bbc77a676bde0d0afe20f40dc9e3e1.socket --xlator-option *replicate*.node-uuid=e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3)
[2014-10-13 17:38:26.944548] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
[2014-10-13 17:38:26.944584] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
[2014-10-13 17:38:26.944689] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-10-13 17:38:26.944701] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-10-13 17:38:26.946667] I [graph.c:254:gf_add_cmdline_options] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: adding option 'node-uuid' for volume 'HA-2TB-TT-Proxmox-cluster-replicate-0' with value 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
[2014-10-13 17:38:26.946684] I [graph.c:254:gf_add_cmdline_options] 0-HA-WIN-TT-1T-replicate-0: adding option 'node-uuid' for volume 'HA-WIN-TT-1T-replicate-0' with value 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
[2014-10-13 17:38:26.948783] I [socket.c:3561:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-1: SSL support is NOT enabled
[2014-10-13 17:38:26.948809] I [socket.c:3576:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-1: using system polling thread
[2014-10-13 17:38:26.949118] I [socket.c:3561:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-0: SSL support is NOT enabled
[2014-10-13 17:38:26.949134] I [socket.c:3576:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-0: using system polling thread
[2014-10-13 17:38:26.951698] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-1: SSL support is NOT enabled
[2014-10-13 17:38:26.951715] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-1: using system polling thread
[2014-10-13 17:38:26.951921] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-0: SSL support is NOT enabled
[2014-10-13 17:38:26.951932] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-0: using system polling thread
[2014-10-13 17:38:26.951959] I [client.c:2294:notify] 0-HA-2TB-TT-Proxmox-cluster-client-0: parent translators are ready, attempting connect on transport
[2014-10-13 17:38:26.952612] I [client.c:2294:notify] 0-HA-2TB-TT-Proxmox-cluster-client-1: parent translators are ready, attempting connect on transport
[2014-10-13 17:38:26.952862] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-0: parent translators are ready, attempting connect on transport
[2014-10-13 17:38:26.953447] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-1: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume HA-2TB-TT-Proxmox-cluster-client-0
2: type protocol/client
3: option remote-host stor1
4: option remote-subvolume /exports/HA-2TB-TT-Proxmox-cluster/2TB
5: option transport-type socket
6: option username 59c66122-55c1-4c28-956e-6189fcb1aff5
7: option password 34b79afb-a93c-431b-900a-b688e67cdbc9
8: option ping-timeout 10
9: end-volume
10:
11: volume HA-2TB-TT-Proxmox-cluster-client-1
12: type protocol/client
13: option remote-host stor2
14: option remote-subvolume /exports/HA-2TB-TT-Proxmox-cluster/2TB
15: option transport-type socket
16: option username 59c66122-55c1-4c28-956e-6189fcb1aff5
17: option password 34b79afb-a93c-431b-900a-b688e67cdbc9
18: option ping-timeout 10
19: end-volume
20:
21: volume HA-2TB-TT-Proxmox-cluster-replicate-0
22: type cluster/replicate
23: option node-uuid e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
24: option background-self-heal-count 0
25: option metadata-self-heal on
26: option data-self-heal on
27: option entry-self-heal on
28: option self-heal-daemon on
29: option iam-self-heal-daemon yes
30: subvolumes HA-2TB-TT-Proxmox-cluster-client-0 HA-2TB-TT-Proxmox-cluster-client-1
31: end-volume
32:
33: volume HA-WIN-TT-1T-client-0
34: type protocol/client
35: option remote-host stor1
36: option remote-subvolume /exports/NFS-WIN/1T
37: option transport-type socket
38: option username 101b907c-ff21-47da-8ba6-37e2920691ce
39: option password f4f29094-891f-4241-8736-5e3302ed8bc8
40: option ping-timeout 10
41: end-volume
42:
43: volume HA-WIN-TT-1T-client-1
44: type protocol/client
45: option remote-host stor2
46: option remote-subvolume /exports/NFS-WIN/1T
47: option transport-type socket
48: option username 101b907c-ff21-47da-8ba6-37e2920691ce
49: option password f4f29094-891f-4241-8736-5e3302ed8bc8
50: option ping-timeout 10
51: end-volume
52:
53: volume HA-WIN-TT-1T-replicate-0
54: type cluster/replicate
55: option node-uuid e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
56: option background-self-heal-count 0
57: option metadata-self-heal on
58: option data-self-heal on
59: option entry-self-heal on
60: option self-heal-daemon on
61: option iam-self-heal-daemon yes
62: subvolumes HA-WIN-TT-1T-client-0 HA-WIN-TT-1T-client-1
63: end-volume
64:
65: volume glustershd
66: type debug/io-stats
67: subvolumes HA-2TB-TT-Proxmox-cluster-replicate-0 HA-WIN-TT-1T-replicate-0
68: end-volume
69:
+------------------------------------------------------------------------------+
[2014-10-13 17:38:26.954036] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-2TB-TT-Proxmox-cluster-client-0: changing port to 49159 (from 0)
[2014-10-13 17:38:26.954308] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-0: changing port to 49160 (from 0)
[2014-10-13 17:38:26.954741] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-2TB-TT-Proxmox-cluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:26.954815] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:26.954999] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Connected to
10.250.0.1:49159, attached to remote volume '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
[2014-10-13 17:38:26.955017] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:26.955073] I [afr-common.c:4131:afr_notify] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Subvolume 'HA-2TB-TT-Proxmox-cluster-client-0' came back up; going online.
[2014-10-13 17:38:26.955127] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Server lk version = 1
[2014-10-13 17:38:26.955151] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Connected to
10.250.0.1:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:38:26.955161] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:26.955226] I [afr-common.c:4131:afr_notify] 0-HA-WIN-TT-1T-replicate-0: Subvolume 'HA-WIN-TT-1T-client-0' came back up; going online.
[2014-10-13 17:38:26.955297] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-0: Server lk version = 1
[2014-10-13 17:38:26.955583] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Another crawl is in progress for HA-2TB-TT-Proxmox-cluster-client-0
[2014-10-13 17:38:26.955589] E [afr-self-heald.c:1479:afr_find_child_position] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: getxattr failed on HA-2TB-TT-Proxmox-cluster-client-1 - (Transport endpoint is not connected)
[2014-10-13 17:38:26.955832] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0
[2014-10-13 17:38:26.955858] E [afr-self-heald.c:1479:afr_find_child_position] 0-HA-WIN-TT-1T-replicate-0: getxattr failed on HA-WIN-TT-1T-client-1 - (Transport endpoint is not connected)
[2014-10-13 17:38:26.964913] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-2TB-TT-Proxmox-cluster-client-1: changing port to 49159 (from 0)
[2014-10-13 17:38:26.965553] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-2TB-TT-Proxmox-cluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:26.965794] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Connected to
10.250.0.2:49159, attached to remote volume '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
[2014-10-13 17:38:26.965815] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:26.965968] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Server lk version = 1
[2014-10-13 17:38:26.967510] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Another crawl is in progress for HA-2TB-TT-Proxmox-cluster-client-0
[2014-10-13 17:38:27.971374] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-1: changing port to 49160 (from 0)
[2014-10-13 17:38:27.971940] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:27.975460] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Connected to
10.250.0.2:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:38:27.975481] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:27.976656] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-1: Server lk version = 1
[2014-10-13 17:41:05.390992] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.408292] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.412221] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2014-10-13 17:41:05.417388] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
root@stor1:~#
glusterfshd stor2
[2014-10-13 17:38:28.992891] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/b1494ca4d047df6e8590d7080131908f.socket --xlator-option *replicate*.node-uuid=abf9e3a7-eb91-4273-acdf-876cd6ba1fe3)
[2014-10-13 17:38:28.994439] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
[2014-10-13 17:38:28.994476] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
[2014-10-13 17:38:28.994581] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-10-13 17:38:28.994594] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-10-13 17:38:28.996569] I [graph.c:254:gf_add_cmdline_options] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: adding option 'node-uuid' for volume 'HA-2TB-TT-Proxmox-cluster-replicate-0' with value 'abf9e3a7-eb91-4273-acdf-876cd6ba1fe3'
[2014-10-13 17:38:28.996585] I [graph.c:254:gf_add_cmdline_options] 0-HA-WIN-TT-1T-replicate-0: adding option 'node-uuid' for volume 'HA-WIN-TT-1T-replicate-0' with value 'abf9e3a7-eb91-4273-acdf-876cd6ba1fe3'
[2014-10-13 17:38:28.998463] I [socket.c:3561:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-1: SSL support is NOT enabled
[2014-10-13 17:38:28.998483] I [socket.c:3576:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-1: using system polling thread
[2014-10-13 17:38:28.998695] I [socket.c:3561:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-0: SSL support is NOT enabled
[2014-10-13 17:38:28.998707] I [socket.c:3576:socket_init] 0-HA-2TB-TT-Proxmox-cluster-client-0: using system polling thread
[2014-10-13 17:38:29.000506] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-1: SSL support is NOT enabled
[2014-10-13 17:38:29.000520] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-1: using system polling thread
[2014-10-13 17:38:29.000723] I [socket.c:3561:socket_init] 0-HA-WIN-TT-1T-client-0: SSL support is NOT enabled
[2014-10-13 17:38:29.000734] I [socket.c:3576:socket_init] 0-HA-WIN-TT-1T-client-0: using system polling thread
[2014-10-13 17:38:29.000762] I [client.c:2294:notify] 0-HA-2TB-TT-Proxmox-cluster-client-0: parent translators are ready, attempting connect on transport
[2014-10-13 17:38:29.001064] I [client.c:2294:notify] 0-HA-2TB-TT-Proxmox-cluster-client-1: parent translators are ready, attempting connect on transport
[2014-10-13 17:38:29.001639] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-0: parent translators are ready, attempting connect on transport
[2014-10-13 17:38:29.001877] I [client.c:2294:notify] 0-HA-WIN-TT-1T-client-1: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume HA-2TB-TT-Proxmox-cluster-client-0
2: type protocol/client
3: option remote-host stor1
4: option remote-subvolume /exports/HA-2TB-TT-Proxmox-cluster/2TB
5: option transport-type socket
6: option username 59c66122-55c1-4c28-956e-6189fcb1aff5
7: option password 34b79afb-a93c-431b-900a-b688e67cdbc9
8: option ping-timeout 10
9: end-volume
10:
11: volume HA-2TB-TT-Proxmox-cluster-client-1
12: type protocol/client
13: option remote-host stor2
14: option remote-subvolume /exports/HA-2TB-TT-Proxmox-cluster/2TB
15: option transport-type socket
16: option username 59c66122-55c1-4c28-956e-6189fcb1aff5
17: option password 34b79afb-a93c-431b-900a-b688e67cdbc9
18: option ping-timeout 10
19: end-volume
20:
21: volume HA-2TB-TT-Proxmox-cluster-replicate-0
22: type cluster/replicate
23: option node-uuid abf9e3a7-eb91-4273-acdf-876cd6ba1fe3
24: option background-self-heal-count 0
25: option metadata-self-heal on
26: option data-self-heal on
27: option entry-self-heal on
28: option self-heal-daemon on
29: option iam-self-heal-daemon yes
30: subvolumes HA-2TB-TT-Proxmox-cluster-client-0 HA-2TB-TT-Proxmox-cluster-client-1
31: end-volume
32:
33: volume HA-WIN-TT-1T-client-0
34: type protocol/client
35: option remote-host stor1
36: option remote-subvolume /exports/NFS-WIN/1T
37: option transport-type socket
38: option username 101b907c-ff21-47da-8ba6-37e2920691ce
39: option password f4f29094-891f-4241-8736-5e3302ed8bc8
40: option ping-timeout 10
41: end-volume
42:
43: volume HA-WIN-TT-1T-client-1
44: type protocol/client
45: option remote-host stor2
46: option remote-subvolume /exports/NFS-WIN/1T
47: option transport-type socket
48: option username 101b907c-ff21-47da-8ba6-37e2920691ce
49: option password f4f29094-891f-4241-8736-5e3302ed8bc8
50: option ping-timeout 10
51: end-volume
52:
53: volume HA-WIN-TT-1T-replicate-0
54: type cluster/replicate
55: option node-uuid abf9e3a7-eb91-4273-acdf-876cd6ba1fe3
56: option background-self-heal-count 0
57: option metadata-self-heal on
58: option data-self-heal on
59: option entry-self-heal on
60: option self-heal-daemon on
61: option iam-self-heal-daemon yes
62: subvolumes HA-WIN-TT-1T-client-0 HA-WIN-TT-1T-client-1
63: end-volume
64:
65: volume glustershd
66: type debug/io-stats
67: subvolumes HA-2TB-TT-Proxmox-cluster-replicate-0 HA-WIN-TT-1T-replicate-0
68: end-volume
69:
+------------------------------------------------------------------------------+
[2014-10-13 17:38:29.002743] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-2TB-TT-Proxmox-cluster-client-1: changing port to 49159 (from 0)
[2014-10-13 17:38:29.003027] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-1: changing port to 49160 (from 0)
[2014-10-13 17:38:29.003290] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-2TB-TT-Proxmox-cluster-client-0: changing port to 49159 (from 0)
[2014-10-13 17:38:29.003334] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-0: changing port to 49160 (from 0)
[2014-10-13 17:38:29.003922] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-2TB-TT-Proxmox-cluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:29.004023] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:29.004139] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-2TB-TT-Proxmox-cluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:29.004202] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Connected to
10.250.0.2:49159, attached to remote volume '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
[2014-10-13 17:38:29.004217] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:29.004266] I [afr-common.c:4131:afr_notify] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Subvolume 'HA-2TB-TT-Proxmox-cluster-client-1' came back up; going online.
[2014-10-13 17:38:29.004318] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-10-13 17:38:29.004368] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Connected to
10.250.0.2:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:38:29.004383] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:29.004429] I [afr-common.c:4131:afr_notify] 0-HA-WIN-TT-1T-replicate-0: Subvolume 'HA-WIN-TT-1T-client-1' came back up; going online.
[2014-10-13 17:38:29.004483] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-1: Server lk version = 1
[2014-10-13 17:38:29.004506] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-1: Server lk version = 1
[2014-10-13 17:38:29.004526] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Connected to
10.250.0.1:49159, attached to remote volume '/exports/HA-2TB-TT-Proxmox-cluster/2TB'.
[2014-10-13 17:38:29.004535] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:29.004613] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Connected to
10.250.0.1:49160, attached to remote volume '/exports/NFS-WIN/1T'.
[2014-10-13 17:38:29.004626] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-10-13 17:38:29.004731] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-2TB-TT-Proxmox-cluster-client-0: Server lk version = 1
[2014-10-13 17:38:29.004796] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-0: Server lk version = 1
[2014-10-13 17:38:29.005291] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-1
[2014-10-13 17:38:29.005303] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Another crawl is in progress for HA-2TB-TT-Proxmox-cluster-client-1
[2014-10-13 17:38:29.005443] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-2TB-TT-Proxmox-cluster-replicate-0: Another crawl is in progress for HA-2TB-TT-Proxmox-cluster-client-1
[2014-10-13 17:41:05.427867] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.443271] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.444111] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2014-10-13 17:41:05.444807] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
brick stor2
[2014-10-13 17:38:17.213386] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x462a0) [0x7f343271f2a0] (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12) [0x7f343371db12] (-->/usr/sbin/glusterfsd(glusterfs_handle_terminate+0x15) [0x7f3434790dd5]))) 0-: received signum (15), shutting down
[2014-10-13 17:38:26.957312] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.5.2 (/usr/sbin/glusterfsd -s stor2 --volfile-id HA-WIN-TT-1T.stor2.exports-NFS-WIN-1T -p /var/lib/glusterd/vols/HA-WIN-TT-1T/run/stor2-exports-NFS-WIN-1T.pid -S /var/run/91514691033d00e666bb151f9c771a26.socket --brick-name /exports/NFS-WIN/1T -l /var/log/glusterfs/bricks/exports-NFS-WIN-1T.log --xlator-option *-posix.glusterd-uuid=abf9e3a7-eb91-4273-acdf-876cd6ba1fe3 --brick-port 49160 --xlator-option HA-WIN-TT-1T-server.listen-port=49160)
[2014-10-13 17:38:26.958864] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
[2014-10-13 17:38:26.958899] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
[2014-10-13 17:38:26.959003] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-10-13 17:38:26.959015] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-10-13 17:38:26.961860] I [graph.c:254:gf_add_cmdline_options] 0-HA-WIN-TT-1T-server: adding option 'listen-port' for volume 'HA-WIN-TT-1T-server' with value '49160'
[2014-10-13 17:38:26.961878] I [graph.c:254:gf_add_cmdline_options] 0-HA-WIN-TT-1T-posix: adding option 'glusterd-uuid' for volume 'HA-WIN-TT-1T-posix' with value 'abf9e3a7-eb91-4273-acdf-876cd6ba1fe3'
[2014-10-13 17:38:26.965032] I [rpcsvc.c:2127:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2014-10-13 17:38:26.965075] W [options.c:888:xl_opt_validate] 0-HA-WIN-TT-1T-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2014-10-13 17:38:26.965097] I [socket.c:3561:socket_init] 0-tcp.HA-WIN-TT-1T-server: SSL support is NOT enabled
[2014-10-13 17:38:26.965105] I [socket.c:3576:socket_init] 0-tcp.HA-WIN-TT-1T-server: using system polling thread
[2014-10-13 17:38:26.965602] W [graph.c:329:_log_if_unknown_option] 0-HA-WIN-TT-1T-quota: option 'timeout' is not recognized
Final graph:
+------------------------------------------------------------------------------+
1: volume HA-WIN-TT-1T-posix
2: type storage/posix
3: option glusterd-uuid abf9e3a7-eb91-4273-acdf-876cd6ba1fe3
4: option directory /exports/NFS-WIN/1T
5: option volume-id 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
6: end-volume
7:
8: volume HA-WIN-TT-1T-changelog
9: type features/changelog
10: option changelog-brick /exports/NFS-WIN/1T
11: option changelog-dir /exports/NFS-WIN/1T/.glusterfs/changelogs
12: subvolumes HA-WIN-TT-1T-posix
13: end-volume
14:
15: volume HA-WIN-TT-1T-access-control
16: type features/access-control
17: subvolumes HA-WIN-TT-1T-changelog
18: end-volume
19:
20: volume HA-WIN-TT-1T-locks
21: type features/locks
22: subvolumes HA-WIN-TT-1T-access-control
23: end-volume
24:
25: volume HA-WIN-TT-1T-io-threads
26: type performance/io-threads
27: subvolumes HA-WIN-TT-1T-locks
28: end-volume
29:
30: volume HA-WIN-TT-1T-index
31: type features/index
32: option index-base /exports/NFS-WIN/1T/.glusterfs/indices
33: subvolumes HA-WIN-TT-1T-io-threads
34: end-volume
35:
36: volume HA-WIN-TT-1T-marker
37: type features/marker
38: option volume-uuid 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
39: option timestamp-file /var/lib/glusterd/vols/HA-WIN-TT-1T/marker.tstamp
40: option xtime off
41: option gsync-force-xtime off
42: option quota off
43: subvolumes HA-WIN-TT-1T-index
44: end-volume
45:
46: volume HA-WIN-TT-1T-quota
47: type features/quota
48: option volume-uuid HA-WIN-TT-1T
49: option server-quota off
50: option timeout 0
51: option deem-statfs off
52: subvolumes HA-WIN-TT-1T-marker
53: end-volume
54:
55: volume /exports/NFS-WIN/1T
56: type debug/io-stats
57: option latency-measurement off
58: option count-fop-hits off
59: subvolumes HA-WIN-TT-1T-quota
60: end-volume
61:
62: volume HA-WIN-TT-1T-server
63: type protocol/server
64: option transport.socket.listen-port 49160
65: option rpc-auth.auth-glusterfs on
66: option rpc-auth.auth-unix on
67: option rpc-auth.auth-null on
68: option transport-type tcp
69: option auth.login./exports/NFS-WIN/1T.allow 101b907c-ff21-47da-8ba6-37e2920691ce
70: option auth.login.101b907c-ff21-47da-8ba6-37e2920691ce.password f4f29094-891f-4241-8736-5e3302ed8bc8
71: option auth.addr./exports/NFS-WIN/1T.allow *
72: subvolumes /exports/NFS-WIN/1T
73: end-volume
74:
+------------------------------------------------------------------------------+
[2014-10-13 17:38:27.985048] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from stor1-14362-2014/10/13-17:38:26:938194-HA-WIN-TT-1T-client-1-0-0 (version: 3.5.2)
[2014-10-13 17:38:28.988700] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-1-0-1 (version: 3.5.2)
[2014-10-13 17:38:29.004121] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from stor2-15494-2014/10/13-17:38:28:989227-HA-WIN-TT-1T-client-1-0-0 (version: 3.5.2)
[2014-10-13 17:38:38.515315] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from glstor-cli-23823-2014/10/13-17:37:54:595571-HA-WIN-TT-1T-client-1-0-0 (version: 3.5.2)
[2014-10-13 17:39:09.872223] I [server.c:520:server_rpc_notify] 0-HA-WIN-TT-1T-server: disconnecting connectionfrom glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-1-0-1
[2014-10-13 17:39:09.872299] I [client_t.c:417:gf_client_unref] 0-HA-WIN-TT-1T-server: Shutting down connection glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-1-0-1
[2014-10-13 17:41:05.427810] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.443234] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.445049] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
root@stor2:~#
brick stor1
[2014-10-13 17:38:24.900066] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.5.2 (/usr/sbin/glusterfsd -s stor1 --volfile-id HA-WIN-TT-1T.stor1.exports-NFS-WIN-1T -p /var/lib/glusterd/vols/HA-WIN-TT-1T/run/stor1-exports-NFS-WIN-1T.pid -S /var/run/02580c93278849804f3f34f7ed8314b2.socket --brick-name /exports/NFS-WIN/1T -l /var/log/glusterfs/bricks/exports-NFS-WIN-1T.log --xlator-option *-posix.glusterd-uuid=e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3 --brick-port 49160 --xlator-option HA-WIN-TT-1T-server.listen-port=49160)
[2014-10-13 17:38:24.902022] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
[2014-10-13 17:38:24.902077] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
[2014-10-13 17:38:24.902214] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-10-13 17:38:24.902239] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2014-10-13 17:38:24.906698] I [graph.c:254:gf_add_cmdline_options] 0-HA-WIN-TT-1T-server: adding option 'listen-port' for volume 'HA-WIN-TT-1T-server' with value '49160'
[2014-10-13 17:38:24.906731] I [graph.c:254:gf_add_cmdline_options] 0-HA-WIN-TT-1T-posix: adding option 'glusterd-uuid' for volume 'HA-WIN-TT-1T-posix' with value 'e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3'
[2014-10-13 17:38:24.908378] I [rpcsvc.c:2127:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2014-10-13 17:38:24.908435] W [options.c:888:xl_opt_validate] 0-HA-WIN-TT-1T-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2014-10-13 17:38:24.908472] I [socket.c:3561:socket_init] 0-tcp.HA-WIN-TT-1T-server: SSL support is NOT enabled
[2014-10-13 17:38:24.908485] I [socket.c:3576:socket_init] 0-tcp.HA-WIN-TT-1T-server: using system polling thread
[2014-10-13 17:38:24.909105] W [graph.c:329:_log_if_unknown_option] 0-HA-WIN-TT-1T-quota: option 'timeout' is not recognized
Final graph:
+------------------------------------------------------------------------------+
1: volume HA-WIN-TT-1T-posix
2: type storage/posix
3: option glusterd-uuid e09cbbc2-08a3-4e5b-83b8-48eb11a1c7b3
4: option directory /exports/NFS-WIN/1T
5: option volume-id 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
6: end-volume
7:
8: volume HA-WIN-TT-1T-changelog
9: type features/changelog
10: option changelog-brick /exports/NFS-WIN/1T
11: option changelog-dir /exports/NFS-WIN/1T/.glusterfs/changelogs
12: subvolumes HA-WIN-TT-1T-posix
13: end-volume
14:
15: volume HA-WIN-TT-1T-access-control
16: type features/access-control
17: subvolumes HA-WIN-TT-1T-changelog
18: end-volume
19:
20: volume HA-WIN-TT-1T-locks
21: type features/locks
22: subvolumes HA-WIN-TT-1T-access-control
23: end-volume
24:
25: volume HA-WIN-TT-1T-io-threads
26: type performance/io-threads
27: subvolumes HA-WIN-TT-1T-locks
28: end-volume
29:
30: volume HA-WIN-TT-1T-index
31: type features/index
32: option index-base /exports/NFS-WIN/1T/.glusterfs/indices
33: subvolumes HA-WIN-TT-1T-io-threads
34: end-volume
35:
36: volume HA-WIN-TT-1T-marker
37: type features/marker
38: option volume-uuid 2937ac01-4cba-44a8-8ff8-0161b67f8ee4
39: option timestamp-file /var/lib/glusterd/vols/HA-WIN-TT-1T/marker.tstamp
40: option xtime off
41: option gsync-force-xtime off
42: option quota off
43: subvolumes HA-WIN-TT-1T-index
44: end-volume
45:
46: volume HA-WIN-TT-1T-quota
47: type features/quota
48: option volume-uuid HA-WIN-TT-1T
49: option server-quota off
50: option timeout 0
51: option deem-statfs off
52: subvolumes HA-WIN-TT-1T-marker
53: end-volume
54:
55: volume /exports/NFS-WIN/1T
56: type debug/io-stats
57: option latency-measurement off
58: option count-fop-hits off
59: subvolumes HA-WIN-TT-1T-quota
60: end-volume
61:
62: volume HA-WIN-TT-1T-server
63: type protocol/server
64: option transport.socket.listen-port 49160
65: option rpc-auth.auth-glusterfs on
66: option rpc-auth.auth-unix on
67: option rpc-auth.auth-null on
68: option transport-type tcp
69: option auth.login./exports/NFS-WIN/1T.allow 101b907c-ff21-47da-8ba6-37e2920691ce
70: option auth.login.101b907c-ff21-47da-8ba6-37e2920691ce.password f4f29094-891f-4241-8736-5e3302ed8bc8
71: option auth.addr./exports/NFS-WIN/1T.allow *
72: subvolumes /exports/NFS-WIN/1T
73: end-volume
74:
+------------------------------------------------------------------------------+
[2014-10-13 17:38:25.933796] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-0-0-1 (version: 3.5.2)
[2014-10-13 17:38:26.954924] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from stor1-14362-2014/10/13-17:38:26:938194-HA-WIN-TT-1T-client-0-0-0 (version: 3.5.2)
[2014-10-13 17:38:28.991488] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from stor2-15494-2014/10/13-17:38:28:989227-HA-WIN-TT-1T-client-0-0-0 (version: 3.5.2)
[2014-10-13 17:38:38.502056] I [server-handshake.c:575:server_setvolume] 0-HA-WIN-TT-1T-server: accepted client from glstor-cli-23823-2014/10/13-17:37:54:595571-HA-WIN-TT-1T-client-0-0-0 (version: 3.5.2)
[2014-10-13 17:39:09.858784] I [server.c:520:server_rpc_notify] 0-HA-WIN-TT-1T-server: disconnecting connectionfrom glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-0-0-1
[2014-10-13 17:39:09.858863] I [client_t.c:417:gf_client_unref] 0-HA-WIN-TT-1T-server: Shutting down connection glstor-cli-20753-2014/10/13-11:50:40:959211-HA-WIN-TT-1T-client-0-0-1
[2014-10-13 17:41:05.390918] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.408236] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2014-10-13 17:41:05.414813] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
seems to be the right part of logs :)