Re: File Corruption when adding bricks to live replica volumes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

On 21/01/16 20:03, Krutika Dhananjay wrote:
Also could you please attach glustershd.log files and the client logs?


Ok, I ran the procedure again just to be sure. This time I got 81 shards being healed from the other bricks, but still got 2 bricks being "reverse healed" from the new brick. A image check on the vm file failed  with:

ERROR: I/O error in check_refcounts_l2
qemu-img: Check failed: Input/output error

removing the brick resolved the problems.

glustershd.log from node vng attached (the new brick node), the fuse client log was empty.


thanks,
-- 
Lindsay Mathieson
[2016-01-22 06:06:38.292707] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/806d2b280009991c06f79516b49e50cf.socket --xlator-option *replicate*.node-uuid=69bbab58-107c-4448-acac-0bc5d1ca7a9f)
[2016-01-22 06:06:38.298974] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-01-22 06:06:38.306613] I [graph.c:269:gf_add_cmdline_options] 0-datastore1-replicate-0: adding option 'node-uuid' for volume 'datastore1-replicate-0' with value '69bbab58-107c-4448-acac-0bc5d1ca7a9f'
[2016-01-22 06:06:38.309168] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-01-22 06:06:38.309204] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2016-01-22 06:06:38.309229] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4
[2016-01-22 06:06:38.310168] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-3: parent translators are ready, attempting connect on transport
[2016-01-22 06:06:38.310740] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-4: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume datastore1-client-3
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host vnb.proxmox.softlog
  5:     option remote-subvolume /vmdata/datastore1
  6:     option transport-type socket
  7:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
  8:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
  9:     option filter-O_DIRECT enable
 10:     option event-threads 4
 11: end-volume
 12:  
 13: volume datastore1-client-4
 14:     type protocol/client
 15:     option ping-timeout 42
 16:     option remote-host vna.proxmox.softlog
 17:     option remote-subvolume /vmdata/datastore1
 18:     option transport-type socket
 19:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
 20:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
 21:     option filter-O_DIRECT enable
 22:     option event-threads 4
 23: end-volume
 24:  
 25: volume datastore1-replicate-0
 26:     type cluster/replicate
 27:     option node-uuid 69bbab58-107c-4448-acac-0bc5d1ca7a9f
 28:     option background-self-heal-count 0
 29:     option metadata-self-heal on
 30:     option data-self-heal on
 31:     option entry-self-heal on
 32:     option self-heal-daemon enable
 33:     option data-self-heal-window-size 256
 34:     option data-self-heal-algorithm full
 35:     option eager-lock enable
 36:     option quorum-type auto
 37:     option iam-self-heal-daemon yes
 38:     subvolumes datastore1-client-3 datastore1-client-4
 39: end-volume
 40:  
 41: volume glustershd
 42:     type debug/io-stats
 43:     subvolumes datastore1-replicate-0
 44: end-volume
 45:  
+------------------------------------------------------------------------------+
[2016-01-22 06:06:39.429243] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-3: changing port to 49156 (from 0)
[2016-01-22 06:06:39.430329] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:06:39.430765] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-3: Connected to datastore1-client-3, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:06:39.430792] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:06:39.430866] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-datastore1-replicate-0: Subvolume 'datastore1-client-3' came back up; going online.
[2016-01-22 06:06:39.431056] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-3: Server lk version = 1
[2016-01-22 06:06:39.439839] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-4: changing port to 49154 (from 0)
[2016-01-22 06:06:39.440959] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:06:39.442225] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-4: Connected to datastore1-client-4, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:06:39.442249] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:06:39.442682] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-4: Server lk version = 1
[2016-01-22 06:10:50.715890] W [glusterfsd.c:1236:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f74979450a4] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x56243e8305b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) [0x56243e830429] ) 0-: received signum (15), shutting down
[2016-01-22 06:10:51.724369] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/806d2b280009991c06f79516b49e50cf.socket --xlator-option *replicate*.node-uuid=69bbab58-107c-4448-acac-0bc5d1ca7a9f)
[2016-01-22 06:10:51.730520] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-01-22 06:10:51.732146] I [graph.c:269:gf_add_cmdline_options] 0-datastore1-replicate-0: adding option 'node-uuid' for volume 'datastore1-replicate-0' with value '69bbab58-107c-4448-acac-0bc5d1ca7a9f'
[2016-01-22 06:10:51.734632] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2016-01-22 06:10:51.734643] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-01-22 06:10:51.734677] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4
[2016-01-22 06:10:51.735979] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-3: parent translators are ready, attempting connect on transport
[2016-01-22 06:10:51.736537] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-4: parent translators are ready, attempting connect on transport
[2016-01-22 06:10:51.736945] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-5: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume datastore1-client-3
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host vnb.proxmox.softlog
  5:     option remote-subvolume /vmdata/datastore1
  6:     option transport-type socket
  7:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
  8:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
  9:     option filter-O_DIRECT enable
[2016-01-22 06:10:51.737420] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-3: changing port to 49156 (from 0)
 10:     option event-threads 4
 11: end-volume
 12:  
 13: volume datastore1-client-4
 14:     type protocol/client
 15:     option ping-timeout 42
 16:     option remote-host vna.proxmox.softlog
 17:     option remote-subvolume /vmdata/datastore1
 18:     option transport-type socket
 19:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
 20:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
 21:     option filter-O_DIRECT enable
 22:     option event-threads 4
 23: end-volume
 24:  
 25: volume datastore1-client-5
 26:     type protocol/client
 27:     option ping-timeout 42
 28:     option remote-host vng.proxmox.softlog
 29:     option remote-subvolume /vmdata/datastore1
[2016-01-22 06:10:51.737584] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
 30:     option transport-type socket
 31:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
 32:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
 33:     option filter-O_DIRECT enable
 34:     option event-threads 4
 35: end-volume
 36:  
 37: volume datastore1-replicate-0
 38:     type cluster/replicate
 39:     option node-uuid 69bbab58-107c-4448-acac-0bc5d1ca7a9f
 40:     option background-self-heal-count 0
 41:     option metadata-self-heal on
 42:     option data-self-heal on
 43:     option entry-self-heal on
 44:     option self-heal-daemon enable
 45:     option data-self-heal-window-size 256
 46:     option data-self-heal-algorithm full
 47:     option eager-lock enable
 48:     option quorum-type auto
 49:     option iam-self-heal-daemon yes
 50:     subvolumes datastore1-client-3 datastore1-client-4 datastore1-client-5
 51: end-volume
 52:  
 53: volume glustershd
 54:     type debug/io-stats
 55:     subvolumes datastore1-replicate-0
 56: end-volume
 57:  
+------------------------------------------------------------------------------+
[2016-01-22 06:10:51.738380] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:10:51.738539] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:10:51.738663] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-5: Connected to datastore1-client-5, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:10:51.738683] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-5: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:10:51.738744] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-datastore1-replicate-0: Subvolume 'datastore1-client-5' came back up; going online.
[2016-01-22 06:10:51.738782] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-5: Server lk version = 1
[2016-01-22 06:10:51.738904] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-3: Connected to datastore1-client-3, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:10:51.738923] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:10:51.739141] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-3: Server lk version = 1
[2016-01-22 06:10:51.876165] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2016-01-22 06:10:51.895737] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-4: changing port to 49154 (from 0)
[2016-01-22 06:10:51.896842] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:10:51.897187] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-4: Connected to datastore1-client-4, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:10:51.897208] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:10:51.897456] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-4: Server lk version = 1
[2016-01-22 06:10:51.951590] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-01-22 06:11:07.164595] W [socket.c:588:__socket_rwv] 0-datastore1-client-5: readv on 10.10.10.241:49155 failed (No data available)
[2016-01-22 06:11:07.164668] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-datastore1-client-5: disconnected from datastore1-client-5. Client process will keep trying to connect to glusterd until brick's port is available
[2016-01-22 06:11:17.727633] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:18.727089] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:21.728428] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:21.728949] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:25.729213] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:25.729721] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:29.730044] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:29.730597] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:33.730883] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:33.731442] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:37.731693] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:37.732244] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:41.732463] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:41.732962] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:45.733284] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:45.733839] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:49.734051] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:49.734602] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:53.734866] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:53.735397] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:11:57.735671] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:11:57.736152] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:01.736480] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:01.737027] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:05.737219] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:05.737752] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:09.737992] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:09.738550] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:13.738809] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:13.739370] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:17.739564] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:17.740076] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:21.740401] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:21.740916] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:25.741133] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:25.741682] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:29.741907] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
[2016-01-22 06:12:29.742430] E [socket.c:2278:socket_connect_finish] 0-datastore1-client-5: connection to 10.10.10.241:49155 failed (Connection refused)
[2016-01-22 06:12:32.574823] W [glusterfsd.c:1236:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7efeec1500a4] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5600316df5b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x59) [0x5600316df429] ) 0-: received signum (15), shutting down
[2016-01-22 06:12:33.583333] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/806d2b280009991c06f79516b49e50cf.socket --xlator-option *replicate*.node-uuid=69bbab58-107c-4448-acac-0bc5d1ca7a9f)
[2016-01-22 06:12:33.589565] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-01-22 06:12:33.591139] I [graph.c:269:gf_add_cmdline_options] 0-datastore1-replicate-0: adding option 'node-uuid' for volume 'datastore1-replicate-0' with value '69bbab58-107c-4448-acac-0bc5d1ca7a9f'
[2016-01-22 06:12:33.593559] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-01-22 06:12:33.593612] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2016-01-22 06:12:33.593619] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4
[2016-01-22 06:12:33.594944] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-3: parent translators are ready, attempting connect on transport
[2016-01-22 06:12:33.595521] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-4: parent translators are ready, attempting connect on transport
[2016-01-22 06:12:33.595959] I [MSGID: 114020] [client.c:2118:notify] 0-datastore1-client-5: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume datastore1-client-3
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host vnb.proxmox.softlog
  5:     option remote-subvolume /vmdata/datastore1
  6:     option transport-type socket
  7:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
  8:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
  9:     option filter-O_DIRECT enable
 10:     option event-threads 4
 11: end-volume
 12:  
 13: volume datastore1-client-4
 14:     type protocol/client
 15:     option ping-timeout 42
 16:     option remote-host vna.proxmox.softlog
 17:     option remote-subvolume /vmdata/datastore1
 18:     option transport-type socket
 19:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
 20:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
 21:     option filter-O_DIRECT enable
 22:     option event-threads 4
 23: end-volume
 24:  
 25: volume datastore1-client-5
 26:     type protocol/client
 27:     option ping-timeout 42
 28:     option remote-host vng.proxmox.softlog
 29:     option remote-subvolume /vmdata/datastore1
 30:     option transport-type socket
 31:     option username 63803a68-c258-4a93-bc1f-612cec7c607d
 32:     option password 9747cdbd-7eeb-42f6-bc36-23e8c6d577bc
 33:     option filter-O_DIRECT enable
 34:     option event-threads 4
 35: end-volume
 36:  
 37: volume datastore1-replicate-0
[2016-01-22 06:12:33.596603] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-5: changing port to 49155 (from 0)
 38:     type cluster/replicate
 39:     option node-uuid 69bbab58-107c-4448-acac-0bc5d1ca7a9f
 40:     option background-self-heal-count 0
 41:     option metadata-self-heal on
 42:     option data-self-heal on
 43:     option entry-self-heal on
 44:     option self-heal-daemon enable
 45:     option data-self-heal-window-size 256
 46:     option data-self-heal-algorithm full
 47:     option eager-lock enable
 48:     option quorum-type auto
 49:     option iam-self-heal-daemon yes
 50:     subvolumes datastore1-client-3 datastore1-client-4 datastore1-client-5
 51: end-volume
 52:  
 53: volume glustershd
 54:     type debug/io-stats
 55:     subvolumes datastore1-replicate-0
 56: end-volume
 57:  
+------------------------------------------------------------------------------+
[2016-01-22 06:12:33.597391] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:12:33.597630] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-5: Connected to datastore1-client-5, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:12:33.597652] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-5: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:12:33.597717] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-datastore1-replicate-0: Subvolume 'datastore1-client-5' came back up; going online.
[2016-01-22 06:12:33.597760] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-5: Server lk version = 1
[2016-01-22 06:12:34.699885] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-4: changing port to 49154 (from 0)
[2016-01-22 06:12:34.701062] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:12:34.701452] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-4: Connected to datastore1-client-4, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:12:34.701474] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:12:34.701708] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-4: Server lk version = 1
[2016-01-22 06:12:34.815720] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-datastore1-client-3: changing port to 49156 (from 0)
[2016-01-22 06:12:34.816848] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-datastore1-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-01-22 06:12:34.817362] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-datastore1-client-3: Connected to datastore1-client-3, attached to remote volume '/vmdata/datastore1'.
[2016-01-22 06:12:34.817385] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-datastore1-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2016-01-22 06:12:34.817621] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-datastore1-client-3: Server lk version = 1
[2016-01-22 06:12:57.241434] I [MSGID: 108026] [afr-self-heald.c:571:afr_shd_full_healer] 0-datastore1-replicate-0: starting full sweep on subvol datastore1-client-5
[2016-01-22 06:13:50.268068] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed data selfheal on 5a3dcdd1-8866-47a5-8432-7b625a2806c3. source=0 sinks=2 
[2016-01-22 06:13:50.269619] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-datastore1-replicate-0: performing metadata selfheal on 5a3dcdd1-8866-47a5-8432-7b625a2806c3
[2016-01-22 06:13:50.312106] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed metadata selfheal on 5a3dcdd1-8866-47a5-8432-7b625a2806c3. source=0 sinks=2 
[2016-01-22 06:14:56.834946] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed data selfheal on bd623902-97ff-4861-9ea4-192d39dcbd47. source=0 sinks=2 
[2016-01-22 06:14:56.838893] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-datastore1-replicate-0: performing metadata selfheal on bd623902-97ff-4861-9ea4-192d39dcbd47
[2016-01-22 06:14:56.844593] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed metadata selfheal on bd623902-97ff-4861-9ea4-192d39dcbd47. source=0 sinks=2 
[2016-01-22 06:16:07.559056] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed data selfheal on 72c1ae82-5a29-4bbf-9fe4-1bf3a9754e8e. source=0 sinks=2 
[2016-01-22 06:16:07.563315] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-datastore1-replicate-0: performing metadata selfheal on 72c1ae82-5a29-4bbf-9fe4-1bf3a9754e8e
[2016-01-22 06:16:07.574167] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed metadata selfheal on 72c1ae82-5a29-4bbf-9fe4-1bf3a9754e8e. source=0 sinks=2 
[2016-01-22 06:17:06.684210] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed data selfheal on 4c83a031-b6b9-40b4-bb53-efd2ea32df06. source=0 sinks=2 
[2016-01-22 06:17:06.687526] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-datastore1-replicate-0: performing metadata selfheal on 4c83a031-b6b9-40b4-bb53-efd2ea32df06
[2016-01-22 06:17:06.693113] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed metadata selfheal on 4c83a031-b6b9-40b4-bb53-efd2ea32df06. source=0 sinks=2 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux