If your VMs use ext4 also check this:
HTH,
I asked him what to do for VMs using XFS and he said he could not find a fix (setting to change) for those.
Diego
On Sep 8, 2017 6:19 AM, "Diego Remolina" <dijuremo@xxxxxxxxx> wrote:
The issue of I/O stopping may also be with glusterfsd not being properly killed before rebooting the server.For example in RHEL 7.4 with official Gluster 3.8.4, the glusterd service does *not* stop glusterfsd when you run systemctl stop glusterdSo give this a try on the nose you wish to reboot:1. Stop glusterd2. Check if glusterfsd processes are still running. If they are, use: killall glusterfsdDouble check no more glusterfsd processes are running.3. Now reboot the server, there should be no stopping of I/O on the VMs.I was able to reproduce the stopping I/O behaviour yesterday on server reboots without stopping glusterfsd processes. JoeJulian recommends against changing network ping timeout. We discussed this via IRC recently.DiegoOn Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot@xxxxxxxxx> wrote:This is the qemu log of instance:
[2017-09-08 09:31:48.381077] C
[rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded
in the last 1 seconds, disconnecting.
[2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5)
[2017-09-08 09:31:48.382451] E [MSGID: 114031]
[client-rpc-fops.c:1593:client3_3_finodelk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:31:48.382835] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.627341 (xid=0x27ab6)
[2017-09-08 09:31:48.383293] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.627403 (xid=0x27ab7)
[2017-09-08 09:31:48.383639] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FINODELK(30)) called at 2017-09-08 09:31:45.627444 (xid=0x27ab8)
[2017-09-08 09:31:48.384015] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9)
[2017-09-08 09:31:48.384056] E [MSGID: 114031]
[client-rpc-fops.c:1646:client3_3_entrylk_cbk]
0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
is not connected]
[2017-09-08 09:31:48.384077] E [MSGID: 108007]
[afr-lk-common.c:825:afr_unlock_entrylk_cbk]
0-gv_openstack_1-replicate-0:
/.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on
gv_openstack_1-client
-1 [Transport endpoint is not connected]
[2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(WRITE(13)) called at 2017-09-08 09:31:45.626871 (xid=0x27ab4)
[2017-09-08 09:31:48.388095] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(READ(12)) called at 2017-09-08 09:31:45.628760 (xid=0x27aba)
[2017-09-08 09:31:48.388450] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(READ(12)) called at 2017-09-08 09:31:45.629421 (xid=0x27abb)
[2017-09-08 09:31:48.388757] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.647966 (xid=0x27abc)
[2017-09-08 09:31:48.389158] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.648128 (xid=0x27abd)
[2017-09-08 09:31:48.389502] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e] (--> /lib64/libgfrpc.so.0(rpc_clnt_ connection_cleanup+0x90)[0x7fb cad8d2170] (--> /lib64/libgfrpc.so.0(rpc_clnt_ notify+0x2a0)[0x7fbcad8d2c20] ))))) 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)
[2017-09-08 09:31:48.389838] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
[2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
[2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
[2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
[2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
[2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
[2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
[2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
[2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
[2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
[2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
[2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
[2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fbcadb0 9e8b]
(--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fbcad8d08 ee]
(--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fbcad8d09f e]
(--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb cad8d2170]
(--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
[2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]