On Mon, Dec 7, 2015 at 2:55 PM, Yannick Perret <yannick.perret@xxxxxxxxxxxxx> wrote: > Hello, > > I'm having problems with glusterfs and server/server encryption. > > I have 2 servers (sto1 & sto2) with latest stable version (3.6.7-1 from > gluster repo) on Debian 8.2 (amd64), with one single volume with > replication. > > Without /var/lib/glusterd/secure-access all works as expected. > Enabling encryption requires a little more work before touching /var/lib/gluster/secure-access. I have written a blog post [1] which should help with the steps for getting encryption working with GlusterFS. Please check it out, and see if you've done everything required. [1] https://kshlm.in/network-encryption-in-glusterfs/ > Then I shut down both servers (without any client mounting the volume), > touch /var/lib/glusterd/secure-access on both servers, and start service on > one of the servers: > root@sto2:~# /etc/init.d/glusterfs-server stop > [ ok ] Stopping glusterfs-server (via systemctl): glusterfs-server.service. > > I touch the file: > root@sto2:~# touch /var/lib/glusterd/secure-access > > I start the service (the other server is still down): > root@sto2:~# /etc/init.d/glusterfs-server start > [ ok ] Starting glusterfs-server (via systemctl): glusterfs-server.service. > root@sto2:~# ps aux | grep glus > root 22538 1.3 0.4 402828 18668 ? Ssl 10:07 0:00 > /usr/sbin/glusterd -p /var/run/glusterd.pid > -> it is running. > > I check the pool: > root@sto2:~# gluster pool list > UUID Hostname State > 5fdb629d-886f-43cb-9a71-582051b0dbb2 sto1... Disconnected > 8f51f101-254e-43f9-82a3-ec02591110b5 localhost Connected > > It is what expected at this point. > But now the gluster daemon is dead: > root@sto2:~# gluster pool list > Connection failed. Please check if gluster daemon is operational. > > I can stop and start again the service, and it dies after the 1st command, > whatever the command (tested with 'gluster volume status' which answers > 'Volume HOME is not started' which is the correct state as I stoped the only > volume before activating server/server encryption). > > Note that at this point the other server is still down and no client is > started. > See at the end the "crash log" from the server. > > > I guess it is not the expected behavior, and it is clearly a different > behavior than without server/server encryption. For example if I remove the > secure-access file: > > root@sto2:~# /etc/init.d/glusterfs-server stop > [ ok ] Stopping glusterfs-server (via systemctl): glusterfs-server.service. > root@sto2:~# rm /var/lib/glusterd/secure-access > root@sto2:~# /etc/init.d/glusterfs-server start > [ ok ] Starting glusterfs-server (via systemctl): glusterfs-server.service. > root@sto2:~# gluster pool list > UUID Hostname State > 5fdb629d-886f-43cb-9a71-582051b0dbb2 sto1... Disconnected > 8f51f101-254e-43f9-82a3-ec02591110b5 localhost Connected > > And whatever I do the daemon is still alive and responding. > > > Is this a bug or I missed something needed when moving to server/server > encryption? > > > Moreover if I try to start the other server without performing any action on > the 1st (to prevent crash I have a "ping-pong" crash (start at sto2 then > start at sto1): > root@sto2:~# /etc/init.d/glusterfs-server start > [ ok ] Starting glusterfs-server (via systemctl): glusterfs-server.service. > root@sto1:~# /etc/init.d/glusterfs-server start > [ ok ] Starting glusterfs-server (via systemctl): glusterfs-server.service. > root@sto1:~# gluster pool list > UUID Hostname State > 8f51f101-254e-43f9-82a3-ec02591110b5 sto2.liris.cnrs.fr Disconnected > 5fdb629d-886f-43cb-9a71-582051b0dbb2 localhost Connected > -> here daemon is dead on sto2. Let restart sto2 daemon: > root@sto2:~# /etc/init.d/glusterfs-server restart > [ ok ] Restarting glusterfs-server (via systemctl): > glusterfs-server.service. > root@sto2:~# gluster pool list > UUID Hostname State > 5fdb629d-886f-43cb-9a71-582051b0dbb2 sto1.liris.cnrs.fr Disconnected > 8f51f101-254e-43f9-82a3-ec02591110b5 localhost Connected > -> here daemon is dead on sto1. > root@sto1:~# gluster pool list > Connection failed. Please check if gluster daemon is operational. > > > If I restart both daemons (mostly) at the same time it works fine: > root@sto1:~# /etc/init.d/glusterfs-server restart > [ ok ] Restarting glusterfs-server (via systemctl): > glusterfs-server.service. > root@sto2:~# /etc/init.d/glusterfs-server restart > [ ok ] Restarting glusterfs-server (via systemctl): glusterfs-server.service > root@sto1:~# gluster pool list > UUID Hostname State > 8f51f101-254e-43f9-82a3-ec02591110b5 sto2.liris.cnrs.fr Connected > 5fdb629d-886f-43cb-9a71-582051b0dbb2 localhost Connected > root@sto2:~# gluster pool list > UUID Hostname State > 5fdb629d-886f-43cb-9a71-582051b0dbb2 sto1.liris.cnrs.fr Connected > 8f51f101-254e-43f9-82a3-ec02591110b5 localhost Connected > > > Of course this is not an expected behavior as after a global shutdown > servers may not restart at the same time. Moreover it is a real problem when > shuting down a single server (i.e. for maintenance) as I get again the > "ping-pong" problem. > > > Any help would be appreciate. > > Note : before that these 2 servers were used for testing replicated volumes > (without encryption) without any problem. > > Regards, > -- > Y. > > Log from sto2: > > cat /var/log/glusterfs/etc-glusterfs-glusterd.vol.log > > [2015-12-07 09:09:43.345640] I [MSGID: 100030] [glusterfsd.c:2035:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.7 > (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) > [2015-12-07 09:09:43.352452] I [glusterd.c:1214:init] 0-management: Maximum > allowed open file descriptors set to 65536 > [2015-12-07 09:09:43.352516] I [glusterd.c:1259:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-12-07 09:09:43.359063] I [socket.c:3880:socket_init] > 0-socket.management: SSL support on the I/O path is ENABLED > [2015-12-07 09:09:43.359102] I [socket.c:3883:socket_init] > 0-socket.management: SSL support for glusterd is ENABLED > [2015-12-07 09:09:43.359138] I [socket.c:3900:socket_init] > 0-socket.management: using private polling thread > [2015-12-07 09:09:43.361848] W [rdma.c:4440:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (Aucun > périphérique de ce type) > [2015-12-07 09:09:43.361885] E [rdma.c:4744:init] 0-rdma.management: Failed > to initialize IB Device > [2015-12-07 09:09:43.361902] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-12-07 09:09:43.362023] W [rpcsvc.c:1524:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-12-07 09:09:43.362267] I [socket.c:3883:socket_init] > 0-socket.management: SSL support for glusterd is ENABLED > [2015-12-07 09:09:46.812491] I > [glusterd-store.c:2048:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 30603 > [2015-12-07 09:09:47.192205] I > [glusterd-handler.c:3179:glusterd_friend_add_from_peerinfo] 0-management: > connect returned 0 > [2015-12-07 09:09:47.192321] I [rpc-clnt.c:969:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-12-07 09:09:47.192564] I [socket.c:3880:socket_init] 0-management: SSL > support on the I/O path is ENABLED > [2015-12-07 09:09:47.192585] I [socket.c:3883:socket_init] 0-management: SSL > support for glusterd is ENABLED > [2015-12-07 09:09:47.192601] I [socket.c:3900:socket_init] 0-management: > using private polling thread > [2015-12-07 09:09:47.195831] E [socket.c:3016:socket_connect] 0-management: > connection attempt on failed, (Connexion refusée) > [2015-12-07 09:09:47.196341] I [MSGID: 106004] > [glusterd-handler.c:4398:__glusterd_peer_rpc_notify] 0-management: Peer > 5fdb629d-886f-43cb-9a71-582051b0dbb2, in Peer in Cluster state, has > disconnected from glusterd. > [2015-12-07 09:09:47.196413] E [socket.c:384:ssl_setup_connection] > 0-management: SSL connect error > [2015-12-07 09:09:47.196480] E [socket.c:2386:socket_poller] 0-management: > client setup failed > [2015-12-07 09:09:47.196534] E [glusterd-utils.c:181:glusterd_unlock] > 0-management: Cluster lock not held! > [2015-12-07 09:09:47.196642] I [mem-pool.c:545:mem_pool_destroy] > 0-management: size=588 max=0 total=0 > [2015-12-07 09:09:47.196671] I [mem-pool.c:545:mem_pool_destroy] > 0-management: size=124 max=0 total=0 > [2015-12-07 09:09:47.196787] I [glusterd.c:146:glusterd_uuid_init] > 0-management: retrieved UUID: 8f51f101-254e-43f9-82a3-ec02591110b5 > Final graph: > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option transport.socket.ssl-enabled on > 4: option rpc-auth.auth-glusterfs on > 5: option rpc-auth.auth-unix on > 6: option rpc-auth.auth-null on > 7: option transport.socket.listen-backlog 128 > 8: option ping-timeout 30 > 9: option transport.socket.read-fail-log off > 10: option transport.socket.keepalive-interval 2 > 11: option transport.socket.keepalive-time 10 > 12: option transport-type rdma > 13: option working-directory /var/lib/glusterd > 14: end-volume > 15: > +------------------------------------------------------------------------------+ > [2015-12-07 09:09:50.348636] E [socket.c:2859:socket_connect] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x147)[0x7f1b5a951497] > (--> > /usr/lib/x86_64-linux-gnu/glusterfs/3.6.7/rpc-transport/socket.so(+0x6c32)[0x7f1b545c3c32] > (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_reconnect+0xb9)[0x7f1b5a723469] > (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_timer_proc+0xcd)[0x7f1b5a96b40d] > (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f1b5a0e50a4] ))))) > 0-socket: invalid argument: this->private > [2015-12-07 09:09:53.349724] E [socket.c:2859:socket_connect] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x147)[0x7f1b5a951497] > (--> > /usr/lib/x86_64-linux-gnu/glusterfs/3.6.7/rpc-transport/socket.so(+0x6c32)[0x7f1b545c3c32] > (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_reconnect+0xb9)[0x7f1b5a723469] > (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_timer_proc+0xcd)[0x7f1b5a96b40d] > (--> /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f1b5a0e50a4] ))))) > 0-socket: invalid argument: this->private > [2015-12-07 09:09:55.604061] W > [glusterd-op-sm.c:4073:glusterd_op_modify_op_ctx] 0-management: op_ctx > modification failed > [2015-12-07 09:09:55.604797] I > [glusterd-handler.c:3836:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume HOME > [2015-12-07 09:09:55.605488] E [glusterd-syncop.c:1184:gd_stage_op_phase] > 0-management: Staging of operation 'Volume Status' failed on localhost : > Volume HOME is not started > [2015-12-07 09:09:47.196634] I [MSGID: 106004] > [glusterd-handler.c:4398:__glusterd_peer_rpc_notify] 0-management: Peer > 5fdb629d-886f-43cb-9a71-582051b0dbb2, in Peer in Cluster state, has > disconnected from glusterd. > pending frames: > patchset: git://git.gluster.com/glusterfs.git > signal received: 11 > time of crash: > 2015-12-07 09:09:56 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 3.6.7 > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb1)[0x7f1b5a9522a1] > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f1b5a96919d] > /lib/x86_64-linux-gnu/libc.so.6(+0x35180)[0x7f1b5996e180] > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_connect+0x8)[0x7f1b5a721f48] > /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_reconnect+0xb9)[0x7f1b5a723469] > /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_timer_proc+0xcd)[0x7f1b5a96b40d] > /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f1b5a0e50a4] > /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1b59a1f04d] > --------- > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
