Hello,
I am testing SSL/TLS support in gluster.
I have gluster replica set with 2 node:
# gluster volume info
Volume Name: repofiles
Type: Replicate
Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: dist-int-master03:/storage/gluster_brick_repofiles
Brick2: dist-int-master04:/storage/gluster_brick_repofiles
Options Reconfigured:
performance.readdir-ahead: on
auth.ssl-allow: dist-master03,dist-master04
client.ssl: on
server.ssl: on
If umount all shares, stop all gluster process and try to start glusterd process I found strange behavior on node where I has started first glusterd.
On gluster 3.7.3 it look like output of gluster peer status is disconnected. But the second node show gluster peer status connected. Restart glusterd service on node where I has start first glusterd have solved the problem.
1. # cat /etc/issue
Ubuntu 14.04.2 LTS \n \l
# dpkg -l | grep gluster
ii glusterfs-client 3.7.3-ubuntu1~trusty1 amd64 clustered file-system (client package)
ii glusterfs-common 3.7.3-ubuntu1~trusty1 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.7.3-ubuntu1~trusty1 amd64 clustered file-system (server package)
In log before restart glusterd:
[2015-08-21 13:31:51.474562] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.3 (args: /usr/sbin
/glusterd -p /var/run/glusterd.pid)
[2015-08-21 13:31:51.477772] I [MSGID: 106478] [glusterd.c:1346:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-08-21 13:31:51.477807] I [MSGID: 106479] [glusterd.c:1395:init] 0-management: Using /var/lib/glusterd as working directory
[2015-08-21 13:31:51.480745] I [socket.c:3968:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED
[2015-08-21 13:31:51.480775] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:31:51.480785] I [socket.c:3988:socket_init] 0-socket.management: using private polling thread
[2015-08-21 13:31:51.481924] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2015-08-21 13:31:51.481948] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device
[2015-08-21 13:31:51.481959] W [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2015-08-21 13:31:51.482005] W [rpcsvc.c:1597:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-08-21 13:31:51.482021] E [MSGID: 106243] [glusterd.c:1619:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2015-08-21 13:31:51.482131] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:31:54.273846] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30703
[2015-08-21 13:31:54.506187] I [MSGID: 106498] [glusterd-handler.c:3570:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-08-21 13:31:54.506288] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-08-21 13:31:54.506416] I [socket.c:3968:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2015-08-21 13:31:54.506430] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED
[2015-08-21 13:31:54.506439] I [socket.c:3988:socket_init] 0-management: using private polling thread
[2015-08-21 13:31:54.508248] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument
[2015-08-21 13:31:54.508277] E [socket.c:3019:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2015-08-21 13:31:54.510377] E [socket.c:3075:socket_connect] 0-management: connection attempt on failed, (Connection refused)
[2015-08-21 13:31:54.510826] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer <dist-int-master04.xxx> (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state <Peer in Cluster>, has disconnected from glusterd.
[2015-08-21 13:31:54.510856] E [socket.c:406:ssl_setup_connection] 0-management: SSL connect error
[2015-08-21 13:31:54.510889] E [socket.c:2442:socket_poller] 0-management: client setup failed
[2015-08-21 13:31:54.510885] E [MSGID: 106155] [glusterd-utils.c:199:glusterd_unlock] 0-management: Cluster lock not held!
[2015-08-21 13:31:54.510936] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer <dist-int-master04.xxx> (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state <Peer in Cluster>, has disconnected from glusterd.
[2015-08-21 13:31:54.510960] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=588 max=0 total=0
[2015-08-21 13:31:54.510971] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=124 max=0 total=0
[2015-08-21 13:31:54.516188] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: c90033d0-1253-40bf-b700-5f65648940dd
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option transport.socket.ssl-enabled on
4: option rpc-auth.auth-glusterfs on
5: option rpc-auth.auth-unix on
6: option rpc-auth.auth-null on
7: option rpc-auth-allow-insecure on
8: option transport.socket.listen-backlog 128
9: option ping-timeout 30
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2015-08-21 13:31:54.517254] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-08-21 13:32:05.981534] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:05.987833] I [MSGID: 106163] [glusterd-handshake.c:1198:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30703
[2015-08-21 13:32:05.992637] I [MSGID: 106490] [glusterd-handler.c:2530:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
[2015-08-21 13:32:13.248817] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-08-21 13:32:13.251969] I [MSGID: 106493] [glusterd-handler.c:3771:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dist-int-master04.xxx (0), ret: 0
[2015-08-21 13:32:13.758331] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:13.764531] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:17.734396] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:17.740518] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:20.736654] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:20.742658] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:23.740409] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:23.871236] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:26.743118] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:26.749210] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
In log after restart:
[2015-08-21 13:35:29.934968] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:35:29.941086] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:35:30.581835] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-08-21 13:35:32.937617] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:35:32.943770] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:35:34.699874] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fee4bafc182] -->/usr/sbin/glusterd(g
lusterfs_sigwaiter+0xd5) [0x7fee4c8396f5] -->/usr/sbin/glusterd(cleanup_and_exit+0x69) [0x7fee4c839589] ) 0-: received signum (15), shutting down
[2015-08-21 13:35:34.700241] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2015-08-21 13:35:34.700260] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2015-08-21 13:35:34.700269] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: GlusterD svc cli, Num: 1238463, Ver: 2,
Port: 0
[2015-08-21 13:35:34.700507] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2015-08-21 13:35:34.700527] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2015-08-21 13:35:34.700535] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: Gluster Handshake, Num: 14398633, Ver: 2
, Port: 0
[2015-08-21 13:35:36.711372] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.3 (args: /usr/sbin
/glusterd -p /var/run/glusterd.pid)
[2015-08-21 13:35:36.714487] I [MSGID: 106478] [glusterd.c:1346:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-08-21 13:35:36.714522] I [MSGID: 106479] [glusterd.c:1395:init] 0-management: Using /var/lib/glusterd as working directory
[2015-08-21 13:35:36.717424] I [socket.c:3968:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED
[2015-08-21 13:35:36.717446] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:36.717455] I [socket.c:3988:socket_init] 0-socket.management: using private polling thread
[2015-08-21 13:35:36.718591] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2015-08-21 13:35:36.718616] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device
[2015-08-21 13:35:36.718627] W [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2015-08-21 13:35:36.718672] W [rpcsvc.c:1597:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-08-21 13:35:36.718690] E [MSGID: 106243] [glusterd.c:1619:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2015-08-21 13:35:36.718799] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:38.987950] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30703
[2015-08-21 13:35:39.388369] I [MSGID: 106498] [glusterd-handler.c:3570:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-08-21 13:35:39.388463] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-08-21 13:35:39.388588] I [socket.c:3968:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2015-08-21 13:35:39.388601] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:39.388610] I [socket.c:3988:socket_init] 0-management: using private polling thread
[2015-08-21 13:35:39.390521] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument
[2015-08-21 13:35:39.390542] E [socket.c:3019:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2015-08-21 13:35:39.397966] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: c90033d0-1253-40bf-b700-5f65648940dd
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option transport.socket.ssl-enabled on
4: option rpc-auth.auth-glusterfs on
5: option rpc-auth.auth-unix on
6: option rpc-auth.auth-null on
7: option rpc-auth-allow-insecure on
8: option transport.socket.listen-backlog 128
9: option ping-timeout 30
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2015-08-21 13:35:39.398830] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-08-21 13:35:39.404083] I [socket.c:401:ssl_setup_connection] 0-management: peer CN = dist-master04
[2015-08-21 13:35:42.215075] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-08-21 13:35:45.552171] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:35:45.786872] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894, host: dist-int-master04.xxx, port: 0
[2015-08-21 13:35:45.789585] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2015-08-21 13:35:45.789699] I [socket.c:3971:socket_init] 0-nfs: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.790201] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2015-08-21 13:35:45.798535] W [socket.c:3063:socket_connect] 0-nfs: Ignore failed connection attempt on , (No such file or directory)
[2015-08-21 13:35:45.798614] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2015-08-21 13:35:45.798698] I [socket.c:3971:socket_init] 0-glustershd: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.799560] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2015-08-21 13:35:45.801467] W [socket.c:3063:socket_connect] 0-glustershd: Ignore failed connection attempt on , (No such file or directory)
[2015-08-21 13:35:45.801557] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2015-08-21 13:35:45.801655] I [socket.c:3971:socket_init] 0-quotad: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.802201] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2015-08-21 13:35:45.802267] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2015-08-21 13:35:45.802352] I [socket.c:3971:socket_init] 0-bitd: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.802825] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2015-08-21 13:35:45.802892] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2015-08-21 13:35:45.802976] I [socket.c:3971:socket_init] 0-scrub: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.803458] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2015-08-21 13:35:45.803929] [run.c:190:runner_log] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7ff79c3e7f46] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0xfc)[0x7ff79c439abc] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0x736)[0x7ff7974e3996] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_brick_start+0xfe)[0x7ff7974e8eae] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_restart_bricks+0xed)[0x7ff7974e90ad] ))))) 0-: Starting GlusterFS: /usr/sbin/glusterfsd -s dist-int-master03.xxx --volfile-id repofiles.dist-int-master03.xxx.storage-gluster_brick_repofiles -p /var/lib/glusterd/vols/repofiles/run/dist-int-master03.xxx-storage-gluster_brick_repofiles.pid -S /var/run/gluster/1603c345d3b0bb50423077d097db360b.socket --brick-name /storage/gluster_brick_repofiles -l /var/log/glusterfs/bricks/storage-gluster_brick_repofiles.log --xlator-option *-posix.glusterd-uuid=c90033d0-1253-40bf-b700-5f65648940dd --brick-port 49152 --xlator-option repofiles-server.listen-port=49152
[2015-08-21 13:35:45.806005] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-08-21 13:35:45.806118] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.806698] I [MSGID: 106492] [glusterd-handler.c:2706:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
[2015-08-21 13:35:45.806743] I [MSGID: 106502] [glusterd-handler.c:2751:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-08-21 13:35:45.806851] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-08-21 13:35:45.806966] I [MSGID: 106006] [glusterd-svc-mgmt.c:319:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2015-08-21 13:35:45.807016] W [socket.c:642:__socket_rwv] 0-glustershd: readv on /var/run/gluster/fb1f2970d6c956f3831f6976e5c51a0e.socket failed (Invalid argument)
[2015-08-21 13:35:45.807035] I [MSGID: 106006] [glusterd-svc-mgmt.c:319:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd.
[2015-08-21 13:35:45.812411] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/1603c345d3b0bb50423077d097db360b.socket failed (Invalid argument)
[2015-08-21 13:35:45.817497] I [MSGID: 106005] [glusterd-handler.c:4899:__glusterd_brick_rpc_notify] 0-management: Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles has disconnected from glusterd.
[2015-08-21 13:35:45.817545] I [MSGID: 106163] [glusterd-handshake.c:1198:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30703
[2015-08-21 13:35:45.825665] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.825763] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.825829] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.890352] I [MSGID: 106490] [glusterd-handler.c:2530:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
[2015-08-21 13:35:45.900221] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:35:45.902916] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.942628] I [MSGID: 106493] [glusterd-handler.c:3771:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dist-int-master04.xxx (0), ret: 0
[2015-08-21 13:35:45.948258] I [MSGID: 106143] [glusterd-pmap.c:229:pmap_registry_bind] 0-pmap: adding brick /storage/gluster_brick_repofiles on port 49152
[2015-08-21 13:35:45.949411] I [MSGID: 106493] [glusterd-rpc-ops.c:692:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
On gluster 3.6.4 and ubuntu 12.04 it look like on node where I has started glusterd first gluster peer status return "peer status: failed
Connection failed. Please check if gluster daemon is operational." But glusterd started:
# ps aux | grep gluster
root 731287 0.0 0.0 45740 10432 ? Ss 13:51 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
root 731288 0.0 0.0 158760 12268 ? Ssl 13:51 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
root 731309 0.0 0.0 11100 940 pts/10 S+ 13:51 0:00 grep --color=auto gluster
This error disappeared after start glusterd service on second node.
I am testing SSL/TLS support in gluster.
I have gluster replica set with 2 node:
# gluster volume info
Volume Name: repofiles
Type: Replicate
Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: dist-int-master03:/storage/gluster_brick_repofiles
Brick2: dist-int-master04:/storage/gluster_brick_repofiles
Options Reconfigured:
performance.readdir-ahead: on
auth.ssl-allow: dist-master03,dist-master04
client.ssl: on
server.ssl: on
If umount all shares, stop all gluster process and try to start glusterd process I found strange behavior on node where I has started first glusterd.
On gluster 3.7.3 it look like output of gluster peer status is disconnected. But the second node show gluster peer status connected. Restart glusterd service on node where I has start first glusterd have solved the problem.
1. # cat /etc/issue
Ubuntu 14.04.2 LTS \n \l
# dpkg -l | grep gluster
ii glusterfs-client 3.7.3-ubuntu1~trusty1 amd64 clustered file-system (client package)
ii glusterfs-common 3.7.3-ubuntu1~trusty1 amd64 GlusterFS common libraries and translator modules
ii glusterfs-server 3.7.3-ubuntu1~trusty1 amd64 clustered file-system (server package)
In log before restart glusterd:
[2015-08-21 13:31:51.474562] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.3 (args: /usr/sbin
/glusterd -p /var/run/glusterd.pid)
[2015-08-21 13:31:51.477772] I [MSGID: 106478] [glusterd.c:1346:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-08-21 13:31:51.477807] I [MSGID: 106479] [glusterd.c:1395:init] 0-management: Using /var/lib/glusterd as working directory
[2015-08-21 13:31:51.480745] I [socket.c:3968:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED
[2015-08-21 13:31:51.480775] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:31:51.480785] I [socket.c:3988:socket_init] 0-socket.management: using private polling thread
[2015-08-21 13:31:51.481924] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2015-08-21 13:31:51.481948] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device
[2015-08-21 13:31:51.481959] W [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2015-08-21 13:31:51.482005] W [rpcsvc.c:1597:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-08-21 13:31:51.482021] E [MSGID: 106243] [glusterd.c:1619:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2015-08-21 13:31:51.482131] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:31:54.273846] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30703
[2015-08-21 13:31:54.506187] I [MSGID: 106498] [glusterd-handler.c:3570:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-08-21 13:31:54.506288] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-08-21 13:31:54.506416] I [socket.c:3968:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2015-08-21 13:31:54.506430] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED
[2015-08-21 13:31:54.506439] I [socket.c:3988:socket_init] 0-management: using private polling thread
[2015-08-21 13:31:54.508248] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument
[2015-08-21 13:31:54.508277] E [socket.c:3019:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2015-08-21 13:31:54.510377] E [socket.c:3075:socket_connect] 0-management: connection attempt on failed, (Connection refused)
[2015-08-21 13:31:54.510826] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer <dist-int-master04.xxx> (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state <Peer in Cluster>, has disconnected from glusterd.
[2015-08-21 13:31:54.510856] E [socket.c:406:ssl_setup_connection] 0-management: SSL connect error
[2015-08-21 13:31:54.510889] E [socket.c:2442:socket_poller] 0-management: client setup failed
[2015-08-21 13:31:54.510885] E [MSGID: 106155] [glusterd-utils.c:199:glusterd_unlock] 0-management: Cluster lock not held!
[2015-08-21 13:31:54.510936] I [MSGID: 106004] [glusterd-handler.c:5051:__glusterd_peer_rpc_notify] 0-management: Peer <dist-int-master04.xxx> (<264cf70c-688c-4d67-b835-5ec7b7689894>), in state <Peer in Cluster>, has disconnected from glusterd.
[2015-08-21 13:31:54.510960] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=588 max=0 total=0
[2015-08-21 13:31:54.510971] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=124 max=0 total=0
[2015-08-21 13:31:54.516188] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: c90033d0-1253-40bf-b700-5f65648940dd
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option transport.socket.ssl-enabled on
4: option rpc-auth.auth-glusterfs on
5: option rpc-auth.auth-unix on
6: option rpc-auth.auth-null on
7: option rpc-auth-allow-insecure on
8: option transport.socket.listen-backlog 128
9: option ping-timeout 30
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2015-08-21 13:31:54.517254] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-08-21 13:32:05.981534] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:05.987833] I [MSGID: 106163] [glusterd-handshake.c:1198:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30703
[2015-08-21 13:32:05.992637] I [MSGID: 106490] [glusterd-handler.c:2530:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
[2015-08-21 13:32:13.248817] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-08-21 13:32:13.251969] I [MSGID: 106493] [glusterd-handler.c:3771:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dist-int-master04.xxx (0), ret: 0
[2015-08-21 13:32:13.758331] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:13.764531] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:17.734396] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:17.740518] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:20.736654] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:20.742658] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:23.740409] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:23.871236] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:32:26.743118] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:32:26.749210] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
In log after restart:
[2015-08-21 13:35:29.934968] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:35:29.941086] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:35:30.581835] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-08-21 13:35:32.937617] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:35:32.943770] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:35:34.699874] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fee4bafc182] -->/usr/sbin/glusterd(g
lusterfs_sigwaiter+0xd5) [0x7fee4c8396f5] -->/usr/sbin/glusterd(cleanup_and_exit+0x69) [0x7fee4c839589] ) 0-: received signum (15), shutting down
[2015-08-21 13:35:34.700241] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2015-08-21 13:35:34.700260] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2015-08-21 13:35:34.700269] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: GlusterD svc cli, Num: 1238463, Ver: 2,
Port: 0
[2015-08-21 13:35:34.700507] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2015-08-21 13:35:34.700527] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2015-08-21 13:35:34.700535] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: Gluster Handshake, Num: 14398633, Ver: 2
, Port: 0
[2015-08-21 13:35:36.711372] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.3 (args: /usr/sbin
/glusterd -p /var/run/glusterd.pid)
[2015-08-21 13:35:36.714487] I [MSGID: 106478] [glusterd.c:1346:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-08-21 13:35:36.714522] I [MSGID: 106479] [glusterd.c:1395:init] 0-management: Using /var/lib/glusterd as working directory
[2015-08-21 13:35:36.717424] I [socket.c:3968:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED
[2015-08-21 13:35:36.717446] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:36.717455] I [socket.c:3988:socket_init] 0-socket.management: using private polling thread
[2015-08-21 13:35:36.718591] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2015-08-21 13:35:36.718616] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device
[2015-08-21 13:35:36.718627] W [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2015-08-21 13:35:36.718672] W [rpcsvc.c:1597:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-08-21 13:35:36.718690] E [MSGID: 106243] [glusterd.c:1619:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2015-08-21 13:35:36.718799] I [socket.c:3971:socket_init] 0-socket.management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:38.987950] I [MSGID: 106513] [glusterd-store.c:2036:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30703
[2015-08-21 13:35:39.388369] I [MSGID: 106498] [glusterd-handler.c:3570:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-08-21 13:35:39.388463] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-08-21 13:35:39.388588] I [socket.c:3968:socket_init] 0-management: SSL support on the I/O path is ENABLED
[2015-08-21 13:35:39.388601] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:39.388610] I [socket.c:3988:socket_init] 0-management: using private polling thread
[2015-08-21 13:35:39.390521] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument
[2015-08-21 13:35:39.390542] E [socket.c:3019:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2015-08-21 13:35:39.397966] I [MSGID: 106544] [glusterd.c:156:glusterd_uuid_init] 0-management: retrieved UUID: c90033d0-1253-40bf-b700-5f65648940dd
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option transport.socket.ssl-enabled on
4: option rpc-auth.auth-glusterfs on
5: option rpc-auth.auth-unix on
6: option rpc-auth.auth-null on
7: option rpc-auth-allow-insecure on
8: option transport.socket.listen-backlog 128
9: option ping-timeout 30
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2015-08-21 13:35:39.398830] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-08-21 13:35:39.404083] I [socket.c:401:ssl_setup_connection] 0-management: peer CN = dist-master04
[2015-08-21 13:35:42.215075] I [MSGID: 106487] [glusterd-handler.c:1402:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2015-08-21 13:35:45.552171] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master04
[2015-08-21 13:35:45.786872] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894, host: dist-int-master04.xxx, port: 0
[2015-08-21 13:35:45.789585] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2015-08-21 13:35:45.789699] I [socket.c:3971:socket_init] 0-nfs: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.790201] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2015-08-21 13:35:45.798535] W [socket.c:3063:socket_connect] 0-nfs: Ignore failed connection attempt on , (No such file or directory)
[2015-08-21 13:35:45.798614] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2015-08-21 13:35:45.798698] I [socket.c:3971:socket_init] 0-glustershd: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.799560] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2015-08-21 13:35:45.801467] W [socket.c:3063:socket_connect] 0-glustershd: Ignore failed connection attempt on , (No such file or directory)
[2015-08-21 13:35:45.801557] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2015-08-21 13:35:45.801655] I [socket.c:3971:socket_init] 0-quotad: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.802201] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2015-08-21 13:35:45.802267] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2015-08-21 13:35:45.802352] I [socket.c:3971:socket_init] 0-bitd: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.802825] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2015-08-21 13:35:45.802892] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2015-08-21 13:35:45.802976] I [socket.c:3971:socket_init] 0-scrub: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.803458] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2015-08-21 13:35:45.803929] [run.c:190:runner_log] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x196)[0x7ff79c3e7f46] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0xfc)[0x7ff79c439abc] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0x736)[0x7ff7974e3996] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_brick_start+0xfe)[0x7ff7974e8eae] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/mgmt/glusterd.so(glusterd_restart_bricks+0xed)[0x7ff7974e90ad] ))))) 0-: Starting GlusterFS: /usr/sbin/glusterfsd -s dist-int-master03.xxx --volfile-id repofiles.dist-int-master03.xxx.storage-gluster_brick_repofiles -p /var/lib/glusterd/vols/repofiles/run/dist-int-master03.xxx-storage-gluster_brick_repofiles.pid -S /var/run/gluster/1603c345d3b0bb50423077d097db360b.socket --brick-name /storage/gluster_brick_repofiles -l /var/log/glusterfs/bricks/storage-gluster_brick_repofiles.log --xlator-option *-posix.glusterd-uuid=c90033d0-1253-40bf-b700-5f65648940dd --brick-port 49152 --xlator-option repofiles-server.listen-port=49152
[2015-08-21 13:35:45.806005] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-08-21 13:35:45.806118] I [socket.c:3971:socket_init] 0-management: SSL support for glusterd is ENABLED
[2015-08-21 13:35:45.806698] I [MSGID: 106492] [glusterd-handler.c:2706:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
[2015-08-21 13:35:45.806743] I [MSGID: 106502] [glusterd-handler.c:2751:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-08-21 13:35:45.806851] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/c6f1a58b839ee4334fc0c8731ca06078.socket failed (Invalid argument)
[2015-08-21 13:35:45.806966] I [MSGID: 106006] [glusterd-svc-mgmt.c:319:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2015-08-21 13:35:45.807016] W [socket.c:642:__socket_rwv] 0-glustershd: readv on /var/run/gluster/fb1f2970d6c956f3831f6976e5c51a0e.socket failed (Invalid argument)
[2015-08-21 13:35:45.807035] I [MSGID: 106006] [glusterd-svc-mgmt.c:319:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd.
[2015-08-21 13:35:45.812411] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/1603c345d3b0bb50423077d097db360b.socket failed (Invalid argument)
[2015-08-21 13:35:45.817497] I [MSGID: 106005] [glusterd-handler.c:4899:__glusterd_brick_rpc_notify] 0-management: Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles has disconnected from glusterd.
[2015-08-21 13:35:45.817545] I [MSGID: 106163] [glusterd-handshake.c:1198:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30703
[2015-08-21 13:35:45.825665] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.825763] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.825829] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.890352] I [MSGID: 106490] [glusterd-handler.c:2530:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
[2015-08-21 13:35:45.900221] E [socket.c:2555:socket_poller] 0-socket.management: error in polling loop
[2015-08-21 13:35:45.902916] I [socket.c:401:ssl_setup_connection] 0-socket.management: peer CN = dist-master03
[2015-08-21 13:35:45.942628] I [MSGID: 106493] [glusterd-handler.c:3771:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dist-int-master04.xxx (0), ret: 0
[2015-08-21 13:35:45.948258] I [MSGID: 106143] [glusterd-pmap.c:229:pmap_registry_bind] 0-pmap: adding brick /storage/gluster_brick_repofiles on port 49152
[2015-08-21 13:35:45.949411] I [MSGID: 106493] [glusterd-rpc-ops.c:692:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 264cf70c-688c-4d67-b835-5ec7b7689894
On gluster 3.6.4 and ubuntu 12.04 it look like on node where I has started glusterd first gluster peer status return "peer status: failed
Connection failed. Please check if gluster daemon is operational." But glusterd started:
# ps aux | grep gluster
root 731287 0.0 0.0 45740 10432 ? Ss 13:51 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
root 731288 0.0 0.0 158760 12268 ? Ssl 13:51 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
root 731309 0.0 0.0 11100 940 pts/10 S+ 13:51 0:00 grep --color=auto gluster
This error disappeared after start glusterd service on second node.
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
