Hi, I have a distributed volume which runs on Fedora 26 systems with glusterfs 3.11.2 from gluster.org repos: ---------- [root@taupo ~]# glusterd --version glusterfs 3.11.2 gluster> volume info gv2 Volume Name: gv2 Type: Distribute Volume ID: 6b468f43-3857-4506-917c-7eaaaef9b6ee Status: Started Snapshot Count: 0 Number of Bricks: 6 Transport-type: tcp Bricks: Brick1: kiwi:/srv/gluster/gv2/brick1/gvol Brick2: kiwi:/srv/gluster/gv2/brick2/gvol Brick3: taupo:/srv/gluster/gv2/brick1/gvol Brick4: fox:/srv/gluster/gv2/brick1/gvol Brick5: fox:/srv/gluster/gv2/brick2/gvol Brick6: logan:/srv/gluster/gv2/brick1/gvol Options Reconfigured: performance.readdir-ahead: on nfs.disable: on gluster> volume status gv2 Status of volume: gv2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick kiwi:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1128 Brick kiwi:/srv/gluster/gv2/brick2/gvol 49153 0 Y 1134 Brick taupo:/srv/gluster/gv2/brick1/gvol N/A N/A N N/A Brick fox:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1169 Brick fox:/srv/gluster/gv2/brick2/gvol 49153 0 Y 1175 Brick logan:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1003 ---------- The machine in question is TAUPO which has one brick that refuses to connect to the cluster. All installations were migrated from glusterfs 3.8.14 on Fedora 24: I simply rsync'ed /var/lib/glusterd to new systems. On all other machines glusterd starts fine and all bricks come up. Hence I suspect a race condition somewhere. The glusterd.log file (attached) shows that the brick connects, and then suddenly disconnects from the cluster: ---------- [2017-08-06 03:12:38.536409] I [glusterd-utils.c:5468:glusterd_brick_start] 0-management: discovered already-running brick /srv/gluster/gv2/brick1/gvol [2017-08-06 03:12:38.536414] I [MSGID: 106143] [glusterd-pmap.c:279:pmap_registry_bind] 0-pmap: adding brick /srv/gluster/gv2/brick1/gvol on port 49153 [2017-08-06 03:12:38.536427] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:38.536500] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536616] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.584598] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.599340] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:38.613745] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. ---------- I checked that cluster.brick-multiplex is off. How can I debug this further? Thanks in advance, -- Leonid Isaev
[2017-08-06 03:12:36.559795] I [MSGID: 100030] [glusterfsd.c:2476:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) [2017-08-06 03:12:36.564906] I [MSGID: 106478] [glusterd.c:1422:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-08-06 03:12:36.564936] I [MSGID: 106479] [glusterd.c:1469:init] 0-management: Using /var/lib/glusterd as working directory [2017-08-06 03:12:36.567903] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.11.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2017-08-06 03:12:36.567916] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2017-08-06 03:12:36.567922] W [rpcsvc.c:1660:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-08-06 03:12:36.567929] E [MSGID: 106243] [glusterd.c:1693:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-08-06 03:12:36.568821] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory] [2017-08-06 03:12:36.573114] I [MSGID: 106513] [glusterd-store.c:2193:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30800 [2017-08-06 03:12:36.691076] I [MSGID: 106498] [glusterd-handler.c:3602:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 The message "I [MSGID: 106498] [glusterd-handler.c:3602:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0" repeated 6 times between [2017-08-06 03:12:36.691076] and [2017-08-06 03:12:36.713022] [2017-08-06 03:12:36.713040] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.713057] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.713102] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.715561] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.715573] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.715612] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.716599] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.716610] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.716647] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.717624] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.717634] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.717672] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.718664] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.718673] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.718710] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.719693] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.719703] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.719740] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.720718] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-08-06 03:12:36.720728] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:36.720764] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-08-06 03:12:36.736985] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 4fc4d794-e463-4424-8531-13c639fedfe9 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-08-06 03:12:36.737096] I [MSGID: 101190] [event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-08-06 03:12:38.525090] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9, host: 128.138.140.58, port: 0 [2017-08-06 03:12:38.535858] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2017-08-06 03:12:38.535924] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-08-06 03:12:38.535934] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-08-06 03:12:38.535941] I [MSGID: 106600] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed [2017-08-06 03:12:38.535956] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2017-08-06 03:12:38.536002] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2017-08-06 03:12:38.536008] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-08-06 03:12:38.536023] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2017-08-06 03:12:38.536088] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2017-08-06 03:12:38.536096] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped [2017-08-06 03:12:38.536109] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2017-08-06 03:12:38.536175] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-08-06 03:12:38.536182] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-08-06 03:12:38.536196] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2017-08-06 03:12:38.536257] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-08-06 03:12:38.536267] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-08-06 03:12:38.536409] I [glusterd-utils.c:5468:glusterd_brick_start] 0-management: discovered already-running brick /srv/gluster/gv2/brick1/gvol [2017-08-06 03:12:38.536414] I [MSGID: 106143] [glusterd-pmap.c:279:pmap_registry_bind] 0-pmap: adding brick /srv/gluster/gv2/brick1/gvol on port 49153 [2017-08-06 03:12:38.536427] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-08-06 03:12:38.536500] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2017-08-06 03:12:38.536616] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.584598] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.599340] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:38.613745] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:12:38.613783] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.614144] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:38.622382] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.636536] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.58 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:38.654277] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225, host: 128.138.140.223, port: 0 [2017-08-06 03:12:38.661217] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.719619] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.719700] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.719719] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9 [2017-08-06 03:12:38.719732] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.719749] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.789917] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:38.800810] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.808300] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.223 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:38.821959] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.821982] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.873314] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a, host: takahe.colorado.edu, port: 0 [2017-08-06 03:12:38.880937] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:38.880974] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:38.929723] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225 [2017-08-06 03:12:38.975223] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5, host: 128.138.140.48, port: 0 [2017-08-06 03:12:38.982264] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:38.982651] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:38.993222] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:38.993246] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.061315] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:39.067929] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to takahe.colorado.edu (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.089534] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.089579] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5, host: 128.138.140.227, port: 0 [2017-08-06 03:12:39.100236] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:39.100258] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.151399] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.158128] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.158154] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c, host: 128.138.140.154, port: 0 [2017-08-06 03:12:39.165171] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.185496] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.213247] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.223893] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.48 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.242085] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a [2017-08-06 03:12:39.242115] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.290703] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.290740] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.290751] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b, host: pukeko.colorado.edu, port: 0 [2017-08-06 03:12:39.297783] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.297806] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.350544] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.357462] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5 [2017-08-06 03:12:39.357480] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.364344] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.405649] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.415886] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.426685] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.227 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.443877] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.443916] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.455961] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.154 (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.473703] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.493665] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.518102] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800 [2017-08-06 03:12:39.525093] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5 [2017-08-06 03:12:39.525112] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.576708] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.576744] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c [2017-08-06 03:12:39.576761] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.587450] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to pukeko.colorado.edu (0), ret: 0, op_ret: 0 [2017-08-06 03:12:39.609156] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:39.616031] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2017-08-06 03:12:39.658011] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b [2017-08-06 03:12:41.583404] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:41.602493] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:12:44.578101] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:47.582732] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:50.583198] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:53.584918] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:56.583692] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:12:59.584040] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:02.585104] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:05.583899] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:08.583690] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:11.585503] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:14.585056] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:17.586200] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:20.586345] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:23.586469] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:26.584786] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:29.585524] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:32.587889] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:35.588441] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:38.588356] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:41.589152] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:44.589795] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:47.590841] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:50.590175] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:53.591069] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:56.589404] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:13:59.591044] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:02.592658] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:05.594516] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:08.593540] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:11.592057] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:14.592725] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:17.593480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:20.593690] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:23.593413] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:26.592297] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:29.596041] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:32.595171] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:35.594801] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 38 times between [2017-08-06 03:12:41.602493] and [2017-08-06 03:14:35.612349] [2017-08-06 03:14:38.595965] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:38.612558] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:14:41.594507] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:44.595193] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:47.596333] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:50.595273] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:53.596662] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:56.596728] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:14:59.596660] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:02.598157] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:05.599272] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:08.598349] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:11.597804] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:14.598448] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:17.599280] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:20.600455] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:23.599751] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:26.600506] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:29.601173] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:32.601107] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:35.601696] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:38.602591] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:41.601936] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:44.599341] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:47.603964] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:50.604358] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:53.600733] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:56.601888] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:15:59.605305] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:02.601723] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:05.605308] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:08.605716] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:11.606025] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:12.478137] I [MSGID: 100030] [glusterfsd.c:2476:main] 0-glusterd: Started running glusterd version 3.11.2 (args: glusterd) [2017-08-06 03:16:12.484758] I [MSGID: 106478] [glusterd.c:1422:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-08-06 03:16:12.484797] I [MSGID: 106479] [glusterd.c:1469:init] 0-management: Using /var/lib/glusterd as working directory [2017-08-06 03:16:12.489459] E [socket.c:796:__socket_server_bind] 0-socket.management: binding to failed: Address already in use [2017-08-06 03:16:12.489485] E [socket.c:799:__socket_server_bind] 0-socket.management: Port is already in use [2017-08-06 03:16:12.489503] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: listening on transport failed [2017-08-06 03:16:12.489515] E [MSGID: 106243] [glusterd.c:1686:init] 0-management: creation of listener failed [2017-08-06 03:16:12.489535] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2017-08-06 03:16:12.489546] E [MSGID: 101066] [graph.c:325:glusterfs_graph_init] 0-management: initializing translator failed [2017-08-06 03:16:12.489558] E [MSGID: 101176] [graph.c:696:glusterfs_graph_activate] 0-graph: init failed [2017-08-06 03:16:12.490488] W [glusterfsd.c:1331:cleanup_and_exit] (-->glusterd(glusterfs_volumes_init+0xf7) [0x5604900d02c7] -->glusterd(glusterfs_process_volfp+0x183) [0x5604900d0183] -->glusterd(cleanup_and_exit+0x5a) [0x5604900ccb7a] ) 0-: received signum (-1), shutting down [2017-08-06 03:16:14.605509] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:17.604803] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:20.608876] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:23.603836] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:26.607734] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:29.609353] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:32.609900] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:35.608544] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:14:38.612558] and [2017-08-06 03:16:35.625771] [2017-08-06 03:16:38.609485] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:38.625303] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:16:41.610589] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:44.610563] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:47.610414] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:50.611649] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:53.609751] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:56.611203] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:16:59.613105] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:02.611623] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:05.611041] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:08.611022] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:11.614079] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:14.613345] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:17.616445] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:20.613447] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:23.614405] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:26.615806] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:29.616985] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:32.617296] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:35.617071] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:38.618057] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:41.613480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:44.618631] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:47.618622] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:50.619449] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:53.619548] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:56.619893] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:17:59.620349] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:02.620491] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:05.622153] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:08.618072] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:11.620160] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:14.620953] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:17.623081] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:20.624232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:23.623405] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:26.621556] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:29.622174] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:32.624101] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:35.624159] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:16:38.625303] and [2017-08-06 03:18:35.643628] [2017-08-06 03:18:38.623154] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:38.638874] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:18:41.625459] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:44.625504] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:47.624487] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:50.625552] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:53.627476] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:56.626804] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:18:59.625783] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:02.625682] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:05.628513] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:08.629161] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:11.628698] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:14.627382] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:17.628452] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:20.624656] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:23.629982] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:26.630140] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:29.631511] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:32.631489] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:33.030413] I [MSGID: 106488] [glusterd-handler.c:1547:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-08-06 03:19:35.630630] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:38.628652] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:41.633954] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:44.629301] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:47.634167] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:50.632232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:53.630952] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:56.630006] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:19:56.645139] I [MSGID: 106499] [glusterd-handler.c:4302:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv2 [2017-08-06 03:19:59.635084] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:02.636198] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:05.635633] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:08.636366] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:11.637104] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:14.635935] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:17.638419] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:20.638921] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:23.638394] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:26.637497] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:29.638240] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:32.640077] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:35.639375] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106488] [glusterd-handler.c:1547:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2017-08-06 03:19:33.030413] and [2017-08-06 03:19:33.032367] The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:18:38.638874] and [2017-08-06 03:20:35.657497] [2017-08-06 03:20:38.638998] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:38.655345] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:20:41.639249] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:44.641530] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:47.640078] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:50.639738] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:53.642584] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:56.639418] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:20:59.641479] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:02.641310] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:05.642192] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:08.643265] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:11.645152] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:14.643147] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:17.642636] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:20.644147] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:23.644211] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:26.645538] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:29.646314] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:32.644232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:35.646143] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:38.646014] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:41.647145] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:44.645480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:47.646920] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:50.645961] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:53.648376] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:56.647314] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:21:59.647240] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:02.649525] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:05.650677] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:08.649927] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:11.651573] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:14.650324] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:17.651756] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:20.648295] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:23.652112] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:26.651214] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:29.649616] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:32.653143] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:35.651358] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:20:38.655345] and [2017-08-06 03:22:35.667266] [2017-08-06 03:22:38.653869] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:38.670345] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd. [2017-08-06 03:22:41.654410] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:44.652675] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:47.654514] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:50.654210] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:53.654301] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:56.655478] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:22:59.654460] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:02.657118] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:05.655457] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:08.656861] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:11.656538] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:14.657619] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:17.659736] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:20.655873] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:23.658645] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:26.657419] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:29.659674] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:32.659161] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:35.660123] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:38.660054] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:41.660019] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:44.659646] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:47.660895] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:50.661367] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:53.663192] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:56.662402] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:23:59.661518] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:02.663579] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:05.664634] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:08.663792] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:11.664684] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:14.663878] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:17.665416] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:20.664696] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:23.662762] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now [2017-08-06 03:24:26.663935] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 36 times between [2017-08-06 03:22:38.670345] and [2017-08-06 03:24:26.680441] [2017-08-06 03:24:28.820960] W [glusterfsd.c:1331:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x736d) [0x7f8adc8c036d] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xf5) [0x55ceb1ffcd85] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a) [0x55ceb1ffcb7a] ) 0-: received signum (15), shutting down
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
