Hello Rafi,
a few days ago I had the scenario described below. Here are the log entries. As you can see glusterd was stopped at 15:22:48. The same time FUSE of shared_storage was stopped. It seems to be nearly the same error message. You can also see that I restarted glusterd manually but FUSE of shared_storage was not restarted
#-> /var/log/glusterfs/glusterd.log
[2018-11-21 15:22:37.406032] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:21:45.377310] and [2018-11-21 15:22:39.382300]
[2018-11-21 15:22:48.623951] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7fb1e5f11559] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x561c3626340d] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a) [0x561c362631fa] ) 0-: received signum (15), shutting down
[2018-11-21 15:22:53.491120] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 4.1.5 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2018-11-21 15:22:53.494326] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536
[2018-11-21 15:22:53.494372] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory
[2018-11-21 15:22:53.494393] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory
[2018-11-21 15:22:53.496564] I [socket.c:4477:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED
[2018-11-21 15:22:53.496598] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:53.496607] I [socket.c:4497:socket_init] 0-socket.management: using private polling thread
[2018-11-21 15:22:53.496615] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1
[2018-11-21 15:22:53.496891] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:53.499933] W [MSGID: 103071] [rdma.c:4629:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2018-11-21 15:22:53.499970] W [MSGID: 103055] [rdma.c:4938:init] 0-rdma.management: Failed to initialize IB Device
[2018-11-21 15:22:53.499983] W [rpc-transport.c:351:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2018-11-21 15:22:53.500148] W [rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2018-11-21 15:22:53.500174] E [MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2018-11-21 15:22:53.500361] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:53.500373] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1
[2018-11-21 15:22:53.500615] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:55.353929] I [MSGID: 106513] [glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 40100
[2018-11-21 15:22:55.423433] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 7e7fd30a-94c9-4c21-9c4b-3817508ef121
[2018-11-21 15:22:55.429214] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2018-11-21 15:22:55.429317] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2018-11-21 15:22:55.429379] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-11-21 15:22:55.429443] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:22:55.429623] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2018-11-21 15:22:55.429635] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:55.429643] I [socket.c:4497:socket_init] 0-management: using private polling thread
[2018-11-21 15:22:55.429650] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:22:55.429914] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:55.431676] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:22:55.431846] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2018-11-21 15:22:55.431858] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:55.431866] I [socket.c:4497:socket_init] 0-management: using private polling thread
[2018-11-21 15:22:55.431873] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:22:55.432136] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option transport.socket.ssl-enabled on
4: option rpc-auth.auth-glusterfs on
5: option rpc-auth.auth-unix on
6: option rpc-auth.auth-null on
7: option rpc-auth-allow-insecure on
8: option transport.listen-backlog 10
9: option event-threads 1
10: option ping-timeout 0
11: option transport.socket.read-fail-log off
12: option transport.socket.keepalive-interval 2
13: option transport.socket.keepalive-time 10
14: option transport-type rdma
15: option working-directory /var/lib/glusterd
16: end-volume
17:
+------------------------------------------------------------------------------+
[2018-11-21 15:22:55.431634] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-11-21 15:22:55.433711] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-11-21 15:22:58.975934] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153
[2018-11-21 15:22:58.979840] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152
[2018-11-21 15:22:59.462323] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b, host: fs-apollo-c1-n1, port: 0
[2018-11-21 15:22:59.463676] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:22:59.463728] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:22:59.464948] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2018-11-21 15:22:59.465116] I [socket.c:4480:socket_init] 0-nfs: SSL support for glusterd is ENABLED
[2018-11-21 15:22:59.465130] I [socket.c:4507:socket_init] 0-nfs: using certificate depth 1
[2018-11-21 15:22:59.465387] E [socket.c:4554:socket_init] 0-nfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:59.466154] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2018-11-21 15:22:59.466195] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped
[2018-11-21 15:22:59.467179] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2018-11-21 15:22:59.467309] I [socket.c:4480:socket_init] 0-glustershd: SSL support for glusterd is ENABLED
[2018-11-21 15:22:59.467321] I [socket.c:4507:socket_init] 0-glustershd: using certificate depth 1
[2018-11-21 15:22:59.467553] E [socket.c:4554:socket_init] 0-glustershd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:59.473619] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 4143
[2018-11-21 15:22:59.478144] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 127.0.0.1:49149 failed (Success)
[2018-11-21 15:23:00.473882] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped
[2018-11-21 15:23:00.474011] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service
[2018-11-21 15:23:01.478838] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2018-11-21 15:23:01.479039] I [socket.c:4480:socket_init] 0-quotad: SSL support for glusterd is ENABLED
[2018-11-21 15:23:01.479054] I [socket.c:4507:socket_init] 0-quotad: using certificate depth 1
[2018-11-21 15:23:01.479336] E [socket.c:4554:socket_init] 0-quotad: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:01.480664] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-11-21 15:23:01.480711] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is stopped
[2018-11-21 15:23:01.480777] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2018-11-21 15:23:01.480902] I [socket.c:4480:socket_init] 0-bitd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:01.480912] I [socket.c:4507:socket_init] 0-bitd: using certificate depth 1
[2018-11-21 15:23:01.481145] E [socket.c:4554:socket_init] 0-bitd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:01.486112] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping bitd daemon running in pid: 4343
[2018-11-21 15:23:02.486437] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-11-21 15:23:02.486531] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting bitd service
[2018-11-21 15:23:03.492955] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2018-11-21 15:23:03.493131] I [socket.c:4480:socket_init] 0-scrub: SSL support for glusterd is ENABLED
[2018-11-21 15:23:03.493147] I [socket.c:4507:socket_init] 0-scrub: using certificate depth 1
[2018-11-21 15:23:03.493416] E [socket.c:4554:socket_init] 0-scrub: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:03.498042] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping scrub daemon running in pid: 4354
[2018-11-21 15:23:04.498341] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-11-21 15:23:04.498426] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting scrub service
[2018-11-21 15:23:05.503972] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick1/glusterbrick
[2018-11-21 15:23:05.504033] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152
[2018-11-21 15:23:05.504103] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:23:05.504229] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.504241] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:23:05.504505] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.510177] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick2/glusterbrick
[2018-11-21 15:23:05.510210] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153
[2018-11-21 15:23:05.510259] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:23:05.510368] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.510380] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:23:05.510635] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.516357] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /var/lib/glusterd/ss_brick
[2018-11-21 15:23:05.516387] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154
[2018-11-21 15:23:05.516433] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:23:05.516538] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.516550] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:23:05.516795] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.522223] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2018-11-21 15:23:05.522301] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.522308] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1
[2018-11-21 15:23:05.522709] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.523245] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2018-11-21 15:23:05.523364] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.523379] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1
[2018-11-21 15:23:05.523544] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.523993] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2018-11-21 15:23:05.524564] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.524585] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1
[2018-11-21 15:23:05.524804] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.525379] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2018-11-21 15:23:05.525451] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.525458] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1
[2018-11-21 15:23:05.525649] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.526277] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2018-11-21 15:23:05.526348] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.526355] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1
[2018-11-21 15:23:05.526507] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.527193] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2018-11-21 15:23:05.527265] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.527274] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1
[2018-11-21 15:23:05.527437] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.529501] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7, host: fs-apollo-c1-n3, port: 0
[2018-11-21 15:23:05.530466] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.531325] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:23:05.531934] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.532036] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100
[2018-11-21 15:23:05.532812] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154
[2018-11-21 15:23:05.533478] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100
[2018-11-21 15:23:05.534519] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.535549] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.538671] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n1 (0), ret: 0, op_ret: 0
[2018-11-21 15:23:05.541086] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.543763] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n3 (0), ret: 0, op_ret: 0
[2018-11-21 15:23:05.546026] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.547097] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.547154] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:23:05.547765] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.548774] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:23:05.548958] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:31.411509] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:23:33.372957] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:23:33.372957] and [2018-11-21 15:23:33.376516]
[2018-11-21 15:24:25.407528] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:24:27.370775] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:24:27.370775] and [2018-11-21 15:24:27.373796]
[2018-11-21 15:25:19.407885] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:25:21.371482] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:26:13.415390] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:25:21.371482] and [2018-11-21 15:26:15.378777]
[2018-11-21 15:27:07.403685] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:27:09.371804] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:28:02.728185] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:28:01.415055] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:27:09.371804] and [2018-11-21 15:28:03.370301]
[2018-11-21 15:28:55.414118] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:28:57.375508] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:28:55.414118] and [2018-11-21 15:30:43.419479]
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:28:57.375508] and [2018-11-21 15:30:45.383263]
[2018-11-21 15:31:37.415788] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:31:39.375304] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:32:31.415510] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:31:39.375304] and [2018-11-21 15:32:33.375024]
[2018-11-21 15:33:25.407192] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:33:27.379564] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:34:20.746832] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:34:19.411130] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:33:27.379564] and [2018-11-21 15:34:21.368809]
[2018-11-21 15:35:13.415187] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:35:15.371647] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:36:00.958833] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n3. Please check log file for details.
[2018-11-21 15:36:00.960197] W [glusterd-locks.c:622:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x30c73) [0x7f206875dc73] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdb5d3) [0x7f20688085d3] ) 0-management: Lock for archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121
[2018-11-21 15:36:00.960234] E [MSGID: 106118] [glusterd-op-sm.c:4173:glusterd_op_ac_lock] 0-management: Unable to acquire lock for archive1
[2018-11-21 15:36:00.960311] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: -1
[2018-11-21 15:36:00.961494] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n1. Please check log file for details.
[2018-11-21 15:36:00.961603] E [MSGID: 106150] [glusterd-syncop.c:1957:gd_sync_task_begin] 0-management: Locking Peers Failed.
[2018-11-21 15:36:00.962117] W [glusterd-locks.c:856:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3099f) [0x7f206875d99f] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdba74) [0x7f2068808a74] ) 0-management: Lock owner mismatch. Lock for vol archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121
[2018-11-21 15:36:00.962144] E [MSGID: 106117] [glusterd-op-sm.c:4236:glusterd_op_ac_unlock] 0-management: Unable to release lock for archive1
[2018-11-21 15:36:00.962222] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: 1
[2018-11-21 15:36:00.962406] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n1. Please check log file for details.
[2018-11-21 15:36:00.962973] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n3. Please check log file for details.
[2018-11-21 15:36:00.963033] E [MSGID: 106151] [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s)
[2018-11-21 15:36:00.991734] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0
[2018-11-21 15:36:01.186043] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0
[2018-11-21 15:36:07.416984] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:35:15.371647] and [2018-11-21 15:36:09.383236]
[2018-11-21 15:37:01.410979] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:37:03.383408] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:37:01.410979] and [2018-11-21 15:38:49.410710]
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:37:03.383408] and [2018-11-21 15:38:51.378524]
[2018-11-21 15:39:43.421483] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:39:44.786383] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:39:45.373093] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:40:37.412592] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:39:45.373093] and [2018-11-21 15:40:39.378904]
[2018-11-21 15:41:31.415866] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:41:33.374993] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:42:25.415585] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:41:33.374993] and [2018-11-21 15:42:27.379013]
[2018-11-21 15:43:19.411237] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:43:21.376718] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:44:13.425206] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:43:21.376718] and [2018-11-21 15:44:15.375796]
[2018-11-21 15:45:07.424344] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:45:09.382299] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:46:02.769673] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:46:01.422742] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:45:09.382299] and [2018-11-21 15:46:03.370463]
[2018-11-21 15:46:55.426992] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:46:57.379488] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
#-> /var/log/glusterfs/run-gluster-shared_storage.log
[2018-11-21 15:22:48.623969] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7f9e25569559] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x561fd19c840d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x5a) [0x561fd19c81fa] ) 0-: received signum (15), shutting down
[2018-11-21 15:22:37.406032] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:21:45.377310] and [2018-11-21 15:22:39.382300]
[2018-11-21 15:22:48.623951] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7fb1e5f11559] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x561c3626340d] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a) [0x561c362631fa] ) 0-: received signum (15), shutting down
[2018-11-21 15:22:53.491120] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 4.1.5 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2018-11-21 15:22:53.494326] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536
[2018-11-21 15:22:53.494372] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory
[2018-11-21 15:22:53.494393] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory
[2018-11-21 15:22:53.496564] I [socket.c:4477:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED
[2018-11-21 15:22:53.496598] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:53.496607] I [socket.c:4497:socket_init] 0-socket.management: using private polling thread
[2018-11-21 15:22:53.496615] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1
[2018-11-21 15:22:53.496891] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:53.499933] W [MSGID: 103071] [rdma.c:4629:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2018-11-21 15:22:53.499970] W [MSGID: 103055] [rdma.c:4938:init] 0-rdma.management: Failed to initialize IB Device
[2018-11-21 15:22:53.499983] W [rpc-transport.c:351:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2018-11-21 15:22:53.500148] W [rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2018-11-21 15:22:53.500174] E [MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2018-11-21 15:22:53.500361] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:53.500373] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1
[2018-11-21 15:22:53.500615] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:55.353929] I [MSGID: 106513] [glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 40100
[2018-11-21 15:22:55.423433] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 7e7fd30a-94c9-4c21-9c4b-3817508ef121
[2018-11-21 15:22:55.429214] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2018-11-21 15:22:55.429317] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2018-11-21 15:22:55.429379] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-11-21 15:22:55.429443] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:22:55.429623] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2018-11-21 15:22:55.429635] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:55.429643] I [socket.c:4497:socket_init] 0-management: using private polling thread
[2018-11-21 15:22:55.429650] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:22:55.429914] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:55.431676] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:22:55.431846] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2018-11-21 15:22:55.431858] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:22:55.431866] I [socket.c:4497:socket_init] 0-management: using private polling thread
[2018-11-21 15:22:55.431873] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:22:55.432136] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option transport.socket.ssl-enabled on
4: option rpc-auth.auth-glusterfs on
5: option rpc-auth.auth-unix on
6: option rpc-auth.auth-null on
7: option rpc-auth-allow-insecure on
8: option transport.listen-backlog 10
9: option event-threads 1
10: option ping-timeout 0
11: option transport.socket.read-fail-log off
12: option transport.socket.keepalive-interval 2
13: option transport.socket.keepalive-time 10
14: option transport-type rdma
15: option working-directory /var/lib/glusterd
16: end-volume
17:
+------------------------------------------------------------------------------+
[2018-11-21 15:22:55.431634] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-11-21 15:22:55.433711] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-11-21 15:22:58.975934] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153
[2018-11-21 15:22:58.979840] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152
[2018-11-21 15:22:59.462323] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b, host: fs-apollo-c1-n1, port: 0
[2018-11-21 15:22:59.463676] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:22:59.463728] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:22:59.464948] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2018-11-21 15:22:59.465116] I [socket.c:4480:socket_init] 0-nfs: SSL support for glusterd is ENABLED
[2018-11-21 15:22:59.465130] I [socket.c:4507:socket_init] 0-nfs: using certificate depth 1
[2018-11-21 15:22:59.465387] E [socket.c:4554:socket_init] 0-nfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:59.466154] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2018-11-21 15:22:59.466195] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped
[2018-11-21 15:22:59.467179] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2018-11-21 15:22:59.467309] I [socket.c:4480:socket_init] 0-glustershd: SSL support for glusterd is ENABLED
[2018-11-21 15:22:59.467321] I [socket.c:4507:socket_init] 0-glustershd: using certificate depth 1
[2018-11-21 15:22:59.467553] E [socket.c:4554:socket_init] 0-glustershd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:22:59.473619] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 4143
[2018-11-21 15:22:59.478144] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 127.0.0.1:49149 failed (Success)
[2018-11-21 15:23:00.473882] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped
[2018-11-21 15:23:00.474011] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service
[2018-11-21 15:23:01.478838] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2018-11-21 15:23:01.479039] I [socket.c:4480:socket_init] 0-quotad: SSL support for glusterd is ENABLED
[2018-11-21 15:23:01.479054] I [socket.c:4507:socket_init] 0-quotad: using certificate depth 1
[2018-11-21 15:23:01.479336] E [socket.c:4554:socket_init] 0-quotad: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:01.480664] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-11-21 15:23:01.480711] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is stopped
[2018-11-21 15:23:01.480777] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2018-11-21 15:23:01.480902] I [socket.c:4480:socket_init] 0-bitd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:01.480912] I [socket.c:4507:socket_init] 0-bitd: using certificate depth 1
[2018-11-21 15:23:01.481145] E [socket.c:4554:socket_init] 0-bitd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:01.486112] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping bitd daemon running in pid: 4343
[2018-11-21 15:23:02.486437] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-11-21 15:23:02.486531] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting bitd service
[2018-11-21 15:23:03.492955] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2018-11-21 15:23:03.493131] I [socket.c:4480:socket_init] 0-scrub: SSL support for glusterd is ENABLED
[2018-11-21 15:23:03.493147] I [socket.c:4507:socket_init] 0-scrub: using certificate depth 1
[2018-11-21 15:23:03.493416] E [socket.c:4554:socket_init] 0-scrub: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:03.498042] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping scrub daemon running in pid: 4354
[2018-11-21 15:23:04.498341] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-11-21 15:23:04.498426] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting scrub service
[2018-11-21 15:23:05.503972] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick1/glusterbrick
[2018-11-21 15:23:05.504033] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152
[2018-11-21 15:23:05.504103] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:23:05.504229] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.504241] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:23:05.504505] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.510177] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick2/glusterbrick
[2018-11-21 15:23:05.510210] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153
[2018-11-21 15:23:05.510259] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:23:05.510368] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.510380] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:23:05.510635] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.516357] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /var/lib/glusterd/ss_brick
[2018-11-21 15:23:05.516387] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154
[2018-11-21 15:23:05.516433] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-11-21 15:23:05.516538] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.516550] I [socket.c:4507:socket_init] 0-management: using certificate depth 1
[2018-11-21 15:23:05.516795] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.522223] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2018-11-21 15:23:05.522301] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.522308] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1
[2018-11-21 15:23:05.522709] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.523245] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2018-11-21 15:23:05.523364] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.523379] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1
[2018-11-21 15:23:05.523544] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.523993] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2018-11-21 15:23:05.524564] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.524585] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1
[2018-11-21 15:23:05.524804] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.525379] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2018-11-21 15:23:05.525451] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.525458] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1
[2018-11-21 15:23:05.525649] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.526277] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2018-11-21 15:23:05.526348] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.526355] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1
[2018-11-21 15:23:05.526507] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.527193] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600
[2018-11-21 15:23:05.527265] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED
[2018-11-21 15:23:05.527274] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1
[2018-11-21 15:23:05.527437] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-11-21 15:23:05.529501] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7, host: fs-apollo-c1-n3, port: 0
[2018-11-21 15:23:05.530466] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.531325] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:23:05.531934] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.532036] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100
[2018-11-21 15:23:05.532812] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154
[2018-11-21 15:23:05.533478] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100
[2018-11-21 15:23:05.534519] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.535549] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.538671] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n1 (0), ret: 0, op_ret: 0
[2018-11-21 15:23:05.541086] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.543763] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n3 (0), ret: 0, op_ret: 0
[2018-11-21 15:23:05.546026] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.547097] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b
[2018-11-21 15:23:05.547154] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:23:05.547765] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:05.548774] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-11-21 15:23:05.548958] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7
[2018-11-21 15:23:31.411509] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:23:33.372957] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:23:33.372957] and [2018-11-21 15:23:33.376516]
[2018-11-21 15:24:25.407528] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:24:27.370775] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:24:27.370775] and [2018-11-21 15:24:27.373796]
[2018-11-21 15:25:19.407885] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:25:21.371482] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:26:13.415390] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:25:21.371482] and [2018-11-21 15:26:15.378777]
[2018-11-21 15:27:07.403685] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:27:09.371804] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:28:02.728185] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:28:01.415055] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:27:09.371804] and [2018-11-21 15:28:03.370301]
[2018-11-21 15:28:55.414118] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:28:57.375508] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:28:55.414118] and [2018-11-21 15:30:43.419479]
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:28:57.375508] and [2018-11-21 15:30:45.383263]
[2018-11-21 15:31:37.415788] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:31:39.375304] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:32:31.415510] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:31:39.375304] and [2018-11-21 15:32:33.375024]
[2018-11-21 15:33:25.407192] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:33:27.379564] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:34:20.746832] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:34:19.411130] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:33:27.379564] and [2018-11-21 15:34:21.368809]
[2018-11-21 15:35:13.415187] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:35:15.371647] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:36:00.958833] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n3. Please check log file for details.
[2018-11-21 15:36:00.960197] W [glusterd-locks.c:622:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x30c73) [0x7f206875dc73] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdb5d3) [0x7f20688085d3] ) 0-management: Lock for archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121
[2018-11-21 15:36:00.960234] E [MSGID: 106118] [glusterd-op-sm.c:4173:glusterd_op_ac_lock] 0-management: Unable to acquire lock for archive1
[2018-11-21 15:36:00.960311] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: -1
[2018-11-21 15:36:00.961494] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n1. Please check log file for details.
[2018-11-21 15:36:00.961603] E [MSGID: 106150] [glusterd-syncop.c:1957:gd_sync_task_begin] 0-management: Locking Peers Failed.
[2018-11-21 15:36:00.962117] W [glusterd-locks.c:856:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3099f) [0x7f206875d99f] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdba74) [0x7f2068808a74] ) 0-management: Lock owner mismatch. Lock for vol archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121
[2018-11-21 15:36:00.962144] E [MSGID: 106117] [glusterd-op-sm.c:4236:glusterd_op_ac_unlock] 0-management: Unable to release lock for archive1
[2018-11-21 15:36:00.962222] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: 1
[2018-11-21 15:36:00.962406] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n1. Please check log file for details.
[2018-11-21 15:36:00.962973] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n3. Please check log file for details.
[2018-11-21 15:36:00.963033] E [MSGID: 106151] [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s)
[2018-11-21 15:36:00.991734] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0
[2018-11-21 15:36:01.186043] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0
[2018-11-21 15:36:07.416984] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:35:15.371647] and [2018-11-21 15:36:09.383236]
[2018-11-21 15:37:01.410979] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:37:03.383408] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
The message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:37:01.410979] and [2018-11-21 15:38:49.410710]
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:37:03.383408] and [2018-11-21 15:38:51.378524]
[2018-11-21 15:39:43.421483] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:39:44.786383] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:39:45.373093] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:40:37.412592] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:39:45.373093] and [2018-11-21 15:40:39.378904]
[2018-11-21 15:41:31.415866] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:41:33.374993] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:42:25.415585] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:41:33.374993] and [2018-11-21 15:42:27.379013]
[2018-11-21 15:43:19.411237] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:43:21.376718] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:44:13.425206] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:43:21.376718] and [2018-11-21 15:44:15.375796]
[2018-11-21 15:45:07.424344] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:45:09.382299] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-11-21 15:46:02.769673] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller 10.10.12.64:49133 failed (Input/output error)
[2018-11-21 15:46:01.422742] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:45:09.382299] and [2018-11-21 15:46:03.370463]
[2018-11-21 15:46:55.426992] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-11-21 15:46:57.379488] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
#-> /var/log/glusterfs/run-gluster-shared_storage.log
[2018-11-21 15:22:48.623969] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7f9e25569559] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x561fd19c840d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x5a) [0x561fd19c81fa] ) 0-: received signum (15), shutting down
[2018-11-21 15:22:48.624042] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/run/gluster/shared_storage'.
[2018-11-21 15:22:48.624185] E [fuse-bridge.c:212:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse device failed: Bad file descriptor
[2018-11-21 15:22:48.624246] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /run/gluster/shared_storage
[2018-11-21 15:22:48.624679] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/run/gluster/shared_storage'.
#-> /var/log/glusterfs/bricks/var-lib-glusterd-ss_brick.log
[2018-11-21 15:22:48.626860] E [socket.c:2792:socket_poller] 0-tcp.gluster_shared_storage-server: socket_poller 10.10.12.62:49151 failed (Success)
[2018-11-21 15:22:48.627048] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:22:48.627214] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:22:48.632303] W [socket.c:599:__socket_rwv] 0-glusterfs: readv on 10.10.12.62:24007 failed (Success)
[2018-11-21 15:22:48.632531] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: fs-apollo-c1-n2
[2018-11-21 15:22:48.632558] I [glusterfsd-mgmt.c:2369:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2018-11-21 15:22:59.478134] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:22:59.478287] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:23:05.529596] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2018-11-21 15:23:05.625165] I [addr.c:55:compare_addr_and_update] 0-/var/lib/glusterd/ss_brick: allowed = "*", received addr = "10.10.12.62"
[2018-11-21 15:23:05.625203] I [login.c:45:gf_auth] 0-auth/login: connecting user name: fs-apollo-c1-n2
[2018-11-21 15:23:05.625238] I [login.c:111:gf_auth] 0-auth/login: allowed user names: *
[2018-11-21 15:23:05.625254] I [MSGID: 115029] [server-handshake.c:763:server_setvolume] 0-gluster_shared_storage-server: accepted client from CTX_ID:1c203d59-02da-46ec-b836-371ebb47e3dc-GRAPH_ID:0-PID:20295-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0 (version: 4.1.5)
[2018-11-21 15:36:01.009586] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-11-21 15:36:01.017077] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2018-11-21 15:36:01.206445] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-11-21 15:36:01.212266] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2018-11-21 15:22:48.624185] E [fuse-bridge.c:212:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse device failed: Bad file descriptor
[2018-11-21 15:22:48.624246] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /run/gluster/shared_storage
[2018-11-21 15:22:48.624679] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/run/gluster/shared_storage'.
#-> /var/log/glusterfs/bricks/var-lib-glusterd-ss_brick.log
[2018-11-21 15:22:48.626860] E [socket.c:2792:socket_poller] 0-tcp.gluster_shared_storage-server: socket_poller 10.10.12.62:49151 failed (Success)
[2018-11-21 15:22:48.627048] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:22:48.627214] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:22:48.632303] W [socket.c:599:__socket_rwv] 0-glusterfs: readv on 10.10.12.62:24007 failed (Success)
[2018-11-21 15:22:48.632531] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: fs-apollo-c1-n2
[2018-11-21 15:22:48.632558] I [glusterfsd-mgmt.c:2369:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2018-11-21 15:22:59.478134] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:22:59.478287] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
[2018-11-21 15:23:05.529596] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2018-11-21 15:23:05.625165] I [addr.c:55:compare_addr_and_update] 0-/var/lib/glusterd/ss_brick: allowed = "*", received addr = "10.10.12.62"
[2018-11-21 15:23:05.625203] I [login.c:45:gf_auth] 0-auth/login: connecting user name: fs-apollo-c1-n2
[2018-11-21 15:23:05.625238] I [login.c:111:gf_auth] 0-auth/login: allowed user names: *
[2018-11-21 15:23:05.625254] I [MSGID: 115029] [server-handshake.c:763:server_setvolume] 0-gluster_shared_storage-server: accepted client from CTX_ID:1c203d59-02da-46ec-b836-371ebb47e3dc-GRAPH_ID:0-PID:20295-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0 (version: 4.1.5)
[2018-11-21 15:36:01.009586] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-11-21 15:36:01.017077] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2018-11-21 15:36:01.206445] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-11-21 15:36:01.212266] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
Am Fr., 9. Nov. 2018 um 11:53 Uhr schrieb David Spisla <spisla80@xxxxxxxxx>:
Sure, I will do that the next time shared_storage is unmount after stopping glusterdAm Fr., 9. Nov. 2018 um 10:23 Uhr schrieb Rafi Kavungal Chundattu Parambil <rkavunga@xxxxxxxxxx>:Hi David Spisla,
gluster_shared_storage works just like yet another volume(There are some special handling related to authentication). So bringing down a glusterd node shouldn't have lead to unmount of the volume.
Can you please share the mount log, brick logs at time when you stopped glusterd. Also make sure your shared storage bricks up and running.
Regards
Rafi KC
----- Original Message -----
From: "David Spisla" <spisla80@xxxxxxxxx>
To: gluster-users@xxxxxxxxxxx
Sent: Tuesday, November 6, 2018 3:25:18 PM
Subject: Shared Storage is unmount after stopping glusterd
Dear Gluster Community,
I have a v4.1.5 4-node-cluster with SLES15 machines. The last time I observe that after stopping glusterd with
$ sudo systemctl stop glusterd
sometimes also the FUSE mount of the shared_storage on the same node is unmount but shared_storage ist still in the volume list. I mean really "sometimes" because it is not reproducible in a deterministic way and I am not sure if there is really a dependency between stopping glusterd and shared_storage.
What is the expected behaviour if stopping glusterd? In my opinion all volumes included shared_storage should still have FUSE mounts.
Regards
David Spisla
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
