Of course. Please find attached. Hope they can shed some light on this. Thanks, Seva 28.04.2017, 12:41, "Mohammed Rafi K C" <rkavunga@xxxxxxxxxx>: > Can you share the glusterd logs from the three nodes ? > > Rafi KC > > On 04/28/2017 02:34 PM, Seva Gluschenko wrote: >> Dear Community, >> >> I call for your wisdom, as it appears that googling for keywords doesn't help much. >> >> I have a glusterfs volume with replica count 2, and I tried to perform the online upgrade procedure described in the docs (http://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/). It all went almost fine when I'd done with the first replica, the only problem was the self-heal procedure that refused to complete until I commented out all IPv6 entries in the /etc/hosts. >> >> So far, being sure that it all should work on the 2nd replica pretty the same as it was on the 1st one, I had proceeded with the upgrade on the replica 2. All of a sudden, it told me that it doesn't see the first replica at all. The state before upgrade was: >> >> sst2# gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> ------------------------------------------------------------------------------ >> Brick sst0:/var/glusterfs 49152 0 Y 3482 >> Brick sst2:/var/glusterfs 49152 0 Y 29863 >> NFS Server on localhost 2049 0 Y 25175 >> Self-heal Daemon on localhost N/A N/A Y 25283 >> NFS Server on sst0 N/A N/A N N/A >> Self-heal Daemon on sst0 N/A N/A Y 4827 >> NFS Server on sst1 N/A N/A N N/A >> Self-heal Daemon on sst1 N/A N/A Y 15009 >> >> Task Status of Volume gv0 >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Peer in Cluster (Connected) >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Sent and Received peer request (Connected) >> >> sst2# gluster volume heal gv0 info >> Brick sst0:/var/glusterfs >> Number of entries: 0 >> >> Brick sst2:/var/glusterfs >> Number of entries: 0 >> >> After upgrade, it looked like this: >> >> sst2# gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> ------------------------------------------------------------------------------ >> Brick sst2:/var/glusterfs N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> >> Task Status of Volume gv0 >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Sent and Received peer request (Connected) >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Peer Rejected (Connected) >> >> My biggest fault probably, at that point I googled and found this article https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Resolving%20Peer%20Rejected/ -- and followed its advice, removing at sst2 all the /var/lib/glusterd contents except the glusterd.info file. As the result, the node, predictably, lost all information about the volume. >> >> sst2# gluster volume status >> No volumes present >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Accepted peer request (Connected) >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Accepted peer request (Connected) >> >> Okay, I thought, this is might be a high time to re-add the brick. Not that easy, Jack: >> >> sst0# gluster volume add-brick gv0 replica 2 'sst2:/var/glusterfs' >> volume add-brick: failed: Operation failed >> >> The reason appeared to be natural: sst0 still knows that there was the replica on sst2. What should I do then? At this point, I tried to recover the volume information on sst2 by putting it offline and copying all the volume info from the sst0. Of course it wasn't enough to just copy as is, I modified /var/lib/glusterd/vols/gv0/sst*\:-var-glusterfs, setting listen-port=0 for the remote brick (sst0) and listen-port=49152 for the local brick (sst2). It didn't help much, unfortunately. The final state I've reached is as follows: >> >> sst2# gluster peer status >> Number of Peers: 2 >> >> Hostname: sst1 >> Uuid: 5a2198de-f536-4328-a278-7f746f276e35 >> State: Sent and Received peer request (Connected) >> >> Hostname: sst0 >> Uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc >> State: Sent and Received peer request (Connected) >> >> sst2# gluster volume info >> >> Volume Name: gv0 >> Type: Replicate >> Volume ID: dd4996c0-04e6-4f9b-a04e-73279c4f112b >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: sst0:/var/glusterfs >> Brick2: sst2:/var/glusterfs >> Options Reconfigured: >> cluster.self-heal-daemon: enable >> performance.readdir-ahead: on >> storage.owner-uid: 1000 >> storage.owner-gid: 1000 >> >> sst2# gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> ------------------------------------------------------------------------------ >> Brick sst2:/var/glusterfs N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> NFS Server on localhost N/A N/A N N/A >> >> Task Status of Volume gv0 >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> Meanwhile, on sst0: >> >> sst0# gluster volume info >> >> Volume Name: gv0 >> Type: Replicate >> Volume ID: dd4996c0-04e6-4f9b-a04e-73279c4f112b >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: sst0:/var/glusterfs >> Brick2: sst2:/var/glusterfs >> Options Reconfigured: >> storage.owner-gid: 1000 >> storage.owner-uid: 1000 >> performance.readdir-ahead: on >> cluster.self-heal-daemon: enable >> >> sst0 ~ # gluster volume status >> Status of volume: gv0 >> Gluster process TCP Port RDMA Port Online Pid >> ------------------------------------------------------------------------------ >> Brick sst0:/var/glusterfs 49152 0 Y 31263 >> NFS Server on localhost N/A N/A N N/A >> Self-heal Daemon on localhost N/A N/A Y 31254 >> >> Task Status of Volume gv0 >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> Any ideas how to bring the sst2 back to normal are appreciated. As a last resort solution, I can schedule the downtime, backup data, kill the volume and start all over, but I would like to know if there is a shorter path. Thank you very much in advance. >> >> -- >> Best Regards, >> >> Seva Gluschenko
[2017-04-28 07:45:46.949444] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:45:46.964849] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:45:46.964879] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:45:46.980085] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:45:46.980117] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:45:46.980126] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:45:46.980174] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:45:46.980331] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:45:51.805596] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:45:51.823515] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:45:51.956771] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:45:51.956944] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:45:51.956964] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:45:51.956991] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:45:51.957487] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:45:51.957481] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:45:51.979336] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:45:51.985947] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:45:51.987908] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:45:51.988045] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 1132736709, remote cksum = 59001066 on peer sst0.webkontrol.ru [2017-04-28 07:45:51.988138] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0.webkontrol.ru (0), ret: 0, op_ret: -1 [2017-04-28 07:45:51.988957] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0.webkontrol.ru, port: 0 [2017-04-28 07:45:51.989872] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:47:46.876416] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:47:51.365167] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 The message "I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0" repeated 2 times between [2017-04-28 07:47:51.365167] and [2017-04-28 07:48:38.114271] [2017-04-28 07:50:36.736022] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:50:53.415851] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:51:03.496294] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7ff1e94456ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55fc1d4f37b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55fc1d4f3624] ) 0-: received signum (15), shutting down [2017-04-28 07:51:45.565862] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:51:45.578720] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:51:45.578750] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:51:45.586456] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:51:45.586494] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:51:45.586502] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:51:45.586547] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:51:45.586559] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:51:50.633962] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:51:50.634079] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 07:51:50.634131] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:51:50.641966] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:51:50.708781] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:50.709243] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:51:50.711011] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:50.711738] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:51:50.712126] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0.webkontrol.ru (24007), ret: 0, op_ret: -1 [2017-04-28 07:51:57.559160] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:51:57.559628] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst0 (24007) [2017-04-28 07:51:57.560342] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:51:57.560368] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:51:57.560840] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:51:57.563115] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0 [2017-04-28 07:51:57.563142] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:51:57.564249] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:52:03.403817] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:53:10.797531] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:53:15.247410] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst1 24007 [2017-04-28 07:53:15.248012] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst1 (24007) [2017-04-28 07:53:15.248718] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:53:15.248740] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:53:15.249100] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:53:15.264998] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1 [2017-04-28 07:53:15.265049] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:53:15.268831] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-28 07:53:18.406851] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:58:57.729300] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fcecfbcf6ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x558bdb9147b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x558bdb914624] ) 0-: received signum (15), shutting down [2017-04-28 08:09:27.689903] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 08:09:27.702947] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 08:09:27.702975] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 08:09:27.706577] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 08:09:27.706614] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 08:09:27.706623] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 08:09:27.706668] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 08:09:27.706680] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 08:09:32.555268] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 08:09:32.658765] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:09:32.658928] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:09:32.658968] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:09:32.658994] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:09:32.659625] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:09:32.659619] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:09:32.659990] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 08:09:32.660357] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 08:09:32.683981] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 08:09:32.693205] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.706749] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.720846] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 08:09:32.720986] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst0.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 08:09:32.730415] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0 [2017-04-28 08:09:32.730457] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 08:09:32.733649] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 08:09:32.743630] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1 [2017-04-28 08:09:32.744075] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 08:09:39.918968] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:10:36.874787] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:19:42.780315] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x5 sent = 2017-04-28 08:09:32.730694. timeout = 600 for sst0:24007 [2017-04-28 08:19:42.780375] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x5 sent = 2017-04-28 08:09:32.752906. timeout = 600 for sst1:24007 [2017-04-28 08:49:13.066613] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f8e90e8c6ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x558a399827b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x558a39982624] ) 0-: received signum (15), shutting down [2017-04-28 08:53:09.957876] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 08:53:09.981856] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 08:53:09.982048] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 08:53:09.989872] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 08:53:09.989909] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 08:53:09.989919] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 08:53:09.989967] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 08:53:09.989980] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 08:53:14.863321] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 08:53:14.922604] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:53:14.922769] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 08:53:14.922809] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:53:14.922836] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:53:14.923680] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 08:53:14.923673] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 08:53:14.924038] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 08:53:14.924394] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 08:53:14.939987] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 08:53:14.942194] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:53:14.953057] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:53:21.072426] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:53:35.596254] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:57:50.550439] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:58:33.322742] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 08:58:33.323176] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 08:59:12.333937] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:28:03.196004] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:28:03.203614] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:28:03.203654] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:28:03.208918] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:28:03.208954] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:28:03.208968] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:28:03.209036] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:28:03.209056] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:28:04.693741] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:28:04.694694] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:28:04.696166] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:28:04.696294] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:28:04.696307] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:28:04.696327] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:28:04.696652] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:28:04.696648] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:28:04.697145] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:28:04.740957] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:04.742308] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 07:28:04.742322] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:28:04.777048] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:28:04.777150] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:28:04.822564] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:28:05.551442] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:28:05.556904] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:05.560526] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:28:05.560596] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 4224731823, remote cksum = 1131418324 on peer sst0 [2017-04-28 07:28:05.560663] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0 (0), ret: 0, op_ret: -1 [2017-04-28 07:28:39.446867] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:28:41.333950] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:28:46.326386] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:28:56.471370] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:32:45.753176] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f2653de26ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x557fad3967b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x557fad396624] ) 0-: received signum (15), shutting down [2017-04-28 07:34:36.198249] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:34:36.205642] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:34:36.205682] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:34:36.211006] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:34:36.211041] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:34:36.211054] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:34:36.211119] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:34:36.211139] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:34:37.795217] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:34:37.797039] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:34:37.799291] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:34:37.799333] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:34:37.799355] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:34:37.800437] E [MSGID: 106187] [glusterd-store.c:4559:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2017-04-28 07:34:37.800479] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2017-04-28 07:34:37.800496] E [MSGID: 101066] [graph.c:325:glusterfs_graph_init] 0-management: initializing translator failed [2017-04-28 07:34:37.800516] E [MSGID: 101176] [graph.c:681:glusterfs_graph_activate] 0-graph: init failed [2017-04-28 07:34:37.800962] W [glusterfsd.c:1329:cleanup_and_exit] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xe5) [0x563af9820185] -->/usr/sbin/glusterd(glusterfs_process_volfp+0x1b7) [0x563af9820047] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x563af981f624] ) 0-: received signum (1), shutting down [2017-04-28 07:36:16.994152] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:36:17.002052] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:36:17.002096] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:36:17.008031] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:36:17.008072] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:36:17.008088] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:36:17.008160] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:36:17.008181] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:36:18.537736] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:36:18.537844] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 07:36:18.537898] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:36:18.538873] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:36:18.545097] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:18.545144] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:36:18.548869] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:36:18.549154] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0.webkontrol.ru (24007), ret: 0, op_ret: -1 [2017-04-28 07:36:19.871627] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:25.519090] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:36:25.519575] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst0 (24007) [2017-04-28 07:36:25.520506] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:36:25.520540] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:36:25.521090] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:36:25.536539] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0 [2017-04-28 07:36:25.536568] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:36:25.540030] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:36:27.783553] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:36:27.784256] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst2 (24007) [2017-04-28 07:36:27.785385] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:36:27.785419] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:36:27.785850] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:36:27.801417] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 07:36:27.801448] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:37:50.228006] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:38:04.539534] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:30.570346] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:36.600198] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:39:40.535823] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:41:40.679683] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:44:36.991808] W [socket.c:593:__socket_rwv] 0-management: readv on sst2:24007 failed (No data available) [2017-04-28 07:44:36.991882] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Probe Sent to Peer>, has disconnected from glusterd. [2017-04-28 07:44:36.992313] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb0a3cc35fb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7fb0a3a90eb1] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb0a3a90fce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7fb0a3a92654] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7fb0a3a93138] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(2)) called at 2017-04-28 07:36:27.801497 (xid=0x5) [2017-04-28 07:44:36.992398] I [socket.c:3550:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1) [2017-04-28 07:44:36.992416] E [rpcsvc.c:1348:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 1) to rpc-transport (socket.management) [2017-04-28 07:44:36.992450] E [MSGID: 106430] [glusterd-utils.c:497:glusterd_submit_reply] 0-glusterd: Reply submission failed [2017-04-28 07:44:48.057303] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:45:51.993206] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:03.529476] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Probe Sent to Peer>, has disconnected from glusterd. [2017-04-28 07:51:15.129289] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:53:15.259543] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:53:15.263540] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:53:15.263668] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:53:15.266626] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:53:15.267540] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-28 07:58:57.733476] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Sent and Received peer request>, has disconnected from glusterd. [2017-04-28 07:59:09.177314] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:09:32.718239] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.727764] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:09:32.727897] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 08:09:32.754625] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:49:13.070794] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Sent and Received peer request>, has disconnected from glusterd. [2017-04-28 08:49:24.601282] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:53:14.954821] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 13:53:11.966640] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-27 13:53:11.969733] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-27 13:53:11.969748] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-27 13:53:11.972015] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-27 13:53:11.972030] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-27 13:53:11.972036] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-27 13:53:11.972065] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-27 13:53:11.972074] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-27 13:53:13.118479] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-27 13:53:13.148442] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-27 13:53:13.150180] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 13:53:13.150312] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 13:53:13.150326] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 13:53:13.150345] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 13:53:13.150800] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-27 13:53:13.150796] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 13:53:13.152013] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-27 13:53:13.687393] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 13:53:13.688892] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 13:53:13.689174] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 13:53:13.693990] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:13.694097] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst1.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-27 13:53:13.694872] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-27 13:53:13.696982] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:16.147785] E [socket.c:2310:socket_connect_finish] 0-management: connection to 2a01:4f8:b15:1023::2:24007 failed (No route to host) [2017-04-27 13:53:16.147844] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-27 13:53:16.148656] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-27 13:53:16.150042] E [socket.c:2310:socket_connect_finish] 0-management: connection to 2a01:4f8:b0a:101b::2:24007 failed (No route to host) [2017-04-27 13:53:16.150088] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-27 13:53:16.151683] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:16.151725] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 13:53:16.985462] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-27 13:53:16.987764] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 13:53:16.987812] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 13:53:16.987901] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 13:53:16.988211] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-04-27 13:53:16.988437] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-04-27 13:53:16.988479] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-04-27 13:53:16.991101] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-04-27 13:53:16.992191] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-04-27 13:53:16.992233] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-04-27 13:53:16.992275] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service [2017-04-27 13:53:17.994617] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-04-27 13:53:17.994907] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-27 13:53:17.994947] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-27 13:53:17.995007] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-04-27 13:53:17.995239] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-27 13:53:17.995270] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-27 13:53:17.995324] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-04-27 13:53:17.995543] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-27 13:53:17.995572] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-27 13:53:17.997781] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 13:53:17.998064] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-04-27 13:53:17.998531] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-27 13:53:17.999620] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:17.999712] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 13:53:18.001367] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-04-27 13:53:18.002771] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd. [2017-04-27 13:53:18.002993] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 13:53:22.012073] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49152 [2017-04-27 13:53:44.017756] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 14:41:27.972234] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0 [2017-04-27 14:41:34.404273] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 14:44:27.822099] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 14:54:08.030062] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fcf739d46ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55bb4de5f7b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55bb4de5f624] ) 0-: received signum (15), shutting down [2017-04-27 14:54:29.766745] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-27 14:54:29.770355] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-27 14:54:29.770374] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-27 14:54:29.772949] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-27 14:54:29.772967] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-27 14:54:29.772974] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-27 14:54:29.773006] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-27 14:54:29.773016] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-27 14:54:30.980463] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-27 14:54:31.015698] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-27 14:54:31.017131] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 14:54:31.017267] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 14:54:31.017281] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 14:54:31.017309] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 14:54:31.017651] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-27 14:54:31.017646] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 14:54:31.018602] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-27 14:54:31.024673] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 14:54:31.028369] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 14:54:31.028858] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-27 14:54:31.029630] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-04-27 14:54:31.029715] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-04-27 14:54:31.029731] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-04-27 14:54:31.030598] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-04-27 14:54:31.030981] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-04-27 14:54:31.030998] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-04-27 14:54:31.031015] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service [2017-04-27 14:54:32.032199] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-04-27 14:54:32.032504] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-27 14:54:32.032543] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-27 14:54:32.032603] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-04-27 14:54:32.032840] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-27 14:54:32.032870] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-27 14:54:32.032923] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-04-27 14:54:32.033141] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-27 14:54:32.033170] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-27 14:54:32.035494] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 14:54:32.035825] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-04-27 14:54:32.038639] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 14:54:32.040678] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-04-27 14:54:32.041627] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd. [2017-04-27 14:54:32.041914] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-27 14:54:32.042781] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 14:54:32.042828] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 14:54:32.045661] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 14:54:32.086515] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-27 14:54:32.094695] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:32.094753] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 14:54:32.096411] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:32.096616] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:32.097496] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-27 14:54:32.099173] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 14:54:36.049986] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49153 [2017-04-27 14:59:37.402947] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 15:00:50.647128] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0 [2017-04-27 15:15:09.894711] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fc8e95e16ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x56485ac0e7b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x56485ac0e624] ) 0-: received signum (15), shutting down [2017-04-27 15:16:51.130160] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-27 15:16:51.134590] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-27 15:16:51.134613] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-27 15:16:51.137835] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-27 15:16:51.137858] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-27 15:16:51.137868] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-27 15:16:51.137910] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-27 15:16:51.137929] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-27 15:16:52.334108] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-27 15:16:52.364194] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-27 15:16:52.365653] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 15:16:52.365782] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-27 15:16:52.365794] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 15:16:52.365813] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 15:16:52.366121] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-27 15:16:52.366117] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-27 15:16:52.366995] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-27 15:16:53.247237] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-27 15:16:53.248077] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-04-27 15:16:53.248177] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-04-27 15:16:53.248195] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-04-27 15:16:53.249126] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-04-27 15:16:53.249534] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-04-27 15:16:53.249551] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-04-27 15:16:53.249569] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service [2017-04-27 15:16:54.251325] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-04-27 15:16:54.251621] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-27 15:16:54.251690] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-27 15:16:54.251751] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-04-27 15:16:54.251984] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-27 15:16:54.252016] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-27 15:16:54.252084] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-04-27 15:16:54.252304] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-27 15:16:54.252334] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-27 15:16:54.254557] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-27 15:16:54.254859] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-04-27 15:16:54.255157] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:54.255227] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 15:16:54.255367] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:54.255459] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-27 15:16:54.257751] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now [2017-04-27 15:16:54.258679] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd. [2017-04-27 15:16:54.259058] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.259195] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.259253] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-04-27 15:16:54.259604] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 15:16:54.260863] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.261834] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-27 15:16:54.263351] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-27 15:16:54.264200] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-27 15:16:54.267379] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:54.268157] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-27 15:16:54.272611] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-27 15:16:58.067807] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 15:16:58.267770] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49152 [2017-04-27 15:17:08.145868] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-27 15:17:11.346625] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:07:08.939198] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:07:22.465221] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0 [2017-04-28 07:07:23.219676] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-28 07:07:23.219740] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-28 07:07:23.219839] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-28 07:07:23.219867] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-28 07:07:23.219959] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-28 07:07:23.219983] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-28 07:07:23.255153] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:07:23.272213] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:26:57.666373] W [socket.c:593:__socket_rwv] 0-management: readv on sst1:24007 failed (No data available) [2017-04-28 07:26:57.666447] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-28 07:26:57.666602] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:26:57.666630] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:27:09.534588] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst1:24007 failed (Connection refused) [2017-04-28 07:28:04.734953] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:04.739439] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:28:05.548023] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 1131418324, remote cksum = 4224731823 on peer sst1 [2017-04-28 07:28:05.548159] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: -1 [2017-04-28 07:28:05.560095] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0 [2017-04-28 07:31:35.775788] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-04-28 07:31:35.775843] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-04-28 07:31:35.775940] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-28 07:31:35.775966] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-28 07:31:35.776055] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-28 07:31:35.776077] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-28 07:31:35.790917] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:31:35.808099] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd [2017-04-28 07:32:19.702837] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:32:45.754974] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer Rejected>, has disconnected from glusterd. [2017-04-28 07:32:45.755166] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:32:45.755197] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:32:57.566578] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst1:24007 failed (Connection refused) [2017-04-28 07:36:18.548429] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 24007 [2017-04-28 07:36:25.529048] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:25.532889] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:36:25.533005] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst1.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:36:25.535830] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:36:25.536507] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0 [2017-04-28 07:37:34.559221] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:52.354497] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:44:36.988131] W [socket.c:593:__socket_rwv] 0-management: readv on sst2:24007 failed (No data available) [2017-04-28 07:44:36.988232] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer in Cluster>, has disconnected from glusterd. [2017-04-28 07:44:36.988390] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:44:36.988420] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:44:48.647919] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:45:51.983723] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:45:51.985981] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:45:51.986141] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 59001066, remote cksum = 1132736709 on peer sst2 [2017-04-28 07:45:51.986269] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: -1 [2017-04-28 07:45:51.987609] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0 [2017-04-28 07:48:15.716415] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:48:28.863531] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:51:03.525725] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer Rejected>, has disconnected from glusterd. [2017-04-28 07:51:03.525969] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:51:03.526019] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:51:13.688993] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:51:50.710054] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 24007 [2017-04-28 07:51:57.559226] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:57.560345] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:51:57.560460] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:51:57.561004] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:51:57.561841] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-28 07:52:31.170428] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:52:56.783783] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:53:42.149638] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:55:03.447810] I [MSGID: 106482] [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management: Received add brick req [2017-04-28 07:55:03.447852] I [MSGID: 106578] [glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management: replica-count is 2 [2017-04-28 07:55:03.447865] E [MSGID: 106291] [glusterd-brick-ops.c:626:__glusterd_handle_add_brick] 0-management: [2017-04-28 07:55:31.925551] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:56:56.035196] I [MSGID: 106482] [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management: Received add brick req [2017-04-28 07:56:56.035247] I [MSGID: 106578] [glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management: replica-count is 2 [2017-04-28 07:56:56.035262] E [MSGID: 106291] [glusterd-brick-ops.c:626:__glusterd_handle_add_brick] 0-management: [2017-04-28 07:58:57.730694] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Accepted peer request>, has disconnected from glusterd. [2017-04-28 07:58:57.730885] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 07:58:57.730916] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 07:59:07.732465] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:09:32.693558] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.718118] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:09:32.718248] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 08:09:32.721359] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 08:09:32.721401] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 08:09:32.731133] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:09:53.596619] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:10:53.987182] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 08:15:03.737152] E [MSGID: 106062] [glusterd-utils.c:9338:glusterd_max_opversion_use_rsp_dict] 0-management: Maximum supported op-version not set in destination dictionary [2017-04-28 08:15:26.819219] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:15:54.288580] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:19:24.708030] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:19:38.912746] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x2f sent = 2017-04-28 08:09:32.721691. timeout = 600 for sst2:24007 [2017-04-28 08:49:13.071592] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Sent and Received peer request>, has disconnected from glusterd. [2017-04-28 08:49:13.071837] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held [2017-04-28 08:49:13.071868] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0 [2017-04-28 08:49:24.046693] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:53:14.944791] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:59:59.771701] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 09:00:35.795605] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 08:59:59.772167] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-04-28 09:01:11.179203] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
