This is exactly the problem,
Systemctl stop glusterd does *not* kill the brick processes.
On CentOS with gluster 3.10.x there is also a service, meant to only stop glusterfsd (brick processes). I think the reboot process may not be properly stopping glusterfsd or network or firewall may be stopped before glusterfsd and so the nodes go into the long timeout.
Once again , in my case a simple reboot hangs my VMs, whereas stopping glusterd *and* glusterfsd processes, then reboot works perfectly.
Diego
On Sep 8, 2017 6:36 AM, "Pavel Szalbot" <pavel.szalbot@xxxxxxxxx> wrote:
Hi Diego,
indeed glusterfsd processes are runnin and it is the reason I do
server reboot instead of systemctl glusterd stop. Is killall different
from reboot in a way glusterfsd processes are terminated in CentOS
(init 1?)?
However I will try this and let you know.
-ps
On Fri, Sep 8, 2017 at 12:19 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
> The issue of I/O stopping may also be with glusterfsd not being properly
> killed before rebooting the server.
>
> For example in RHEL 7.4 with official Gluster 3.8.4, the glusterd service
> does *not* stop glusterfsd when you run systemctl stop glusterd
>
> So give this a try on the nose you wish to reboot:
>
> 1. Stop glusterd
> 2. Check if glusterfsd processes are still running. If they are, use:
> killall glusterfsd
> Double check no more glusterfsd processes are running.
> 3. Now reboot the server, there should be no stopping of I/O on the VMs.
>
> I was able to reproduce the stopping I/O behaviour yesterday on server
> reboots without stopping glusterfsd processes. JoeJulian recommends against
> changing network ping timeout. We discussed this via IRC recently.
>
> Diego
>
> On Sep 8, 2017 5:56 AM, "Pavel Szalbot" <pavel.szalbot@xxxxxxxxx> wrote:
>
> This is the qemu log of instance:
>
> [2017-09-08 09:31:48.381077] C
> [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired]
> 0-gv_openstack_1-client-1: server 10.0.1.202:49152 has not responded
> in the last 1 seconds, disconnecting.
> [2017-09-08 09:31:48.382411] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.626882 (xid=0x27ab5)
> [2017-09-08 09:31:48.382451] E [MSGID: 114031]
> [client-rpc-fops.c:1593:client3_3_finodelk_cbk]
> 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
> is not connected]
> [2017-09-08 09:31:48.382835] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.627341 (xid=0x27ab6)
> [2017-09-08 09:31:48.383293] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.627403 (xid=0x27ab7)
> [2017-09-08 09:31:48.383639] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FINODELK(30)) called at 2017-09-08 09:31:45.627444 (xid=0x27ab8)
> [2017-09-08 09:31:48.384015] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(ENTRYLK(31)) called at 2017-09-08 09:31:45.627464 (xid=0x27ab9)
> [2017-09-08 09:31:48.384056] E [MSGID: 114031]
> [client-rpc-fops.c:1646:client3_3_entrylk_cbk]
> 0-gv_openstack_1-client-1: remote operation failed [Transport endpoint
> is not connected]
> [2017-09-08 09:31:48.384077] E [MSGID: 108007]
> [afr-lk-common.c:825:afr_unlock_entrylk_cbk]
> 0-gv_openstack_1-replicate-0:
> /.shard/fe56e79d-7800-4191-8cad-276545017ce7.4223: unlock failed on
> gv_openstack_1-client
> -1 [Transport endpoint is not connected]
> [2017-09-08 09:31:48.384336] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(WRITE(13)) called at 2017-09-08 09:31:45.626871 (xid=0x27ab4)
> [2017-09-08 09:31:48.388095] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(READ(12)) called at 2017-09-08 09:31:45.628760 (xid=0x27aba)
> [2017-09-08 09:31:48.388450] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(READ(12)) called at 2017-09-08 09:31:45.629421 (xid=0x27abb)
> [2017-09-08 09:31:48.388757] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.647966 (xid=0x27abc)
> [2017-09-08 09:31:48.389158] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.648128 (xid=0x27abd)
> [2017-09-08 09:31:48.389502] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--
>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
>> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
>> op(LOOKUP(27)) called at 2017-09-08 09:31:45.648260 (xid=0x27abe)
> [2017-09-08 09:31:48.389838] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652420 (xid=0x27abf)
> [2017-09-08 09:31:48.390217] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652552 (xid=0x27ac0)
> [2017-09-08 09:31:48.390558] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652678 (xid=0x27ac1)
> [2017-09-08 09:31:48.390886] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652810 (xid=0x27ac2)
> [2017-09-08 09:31:48.391263] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(LOOKUP(27)) called at 2017-09-08 09:31:45.652938 (xid=0x27ac3)
> [2017-09-08 09:31:48.391650] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371223 (xid=0x27ac4)
> [2017-09-08 09:31:48.392007] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371319 (xid=0x27ac5)
> [2017-09-08 09:31:48.392428] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371406 (xid=0x27ac6)
> [2017-09-08 09:31:48.392727] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371470 (xid=0x27ac7)
> [2017-09-08 09:31:48.393029] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371533 (xid=0x27ac8)
> [2017-09-08 09:31:48.393397] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371607 (xid=0x27ac9)
> [2017-09-08 09:31:48.393720] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371676 (xid=0x27aca)
> [2017-09-08 09:31:48.394092] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]
> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[ 0x7fbcad8d08ee]
> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[ 0x7fbcad8d09fe]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[ 0x7fbcad8d2170]
> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fbcad8d2c20] )))))
> 0-gv_openstack_1-client-1: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2017-09-08 09:31:46.371739 (xid=0x27acb)
> [2017-09-08 09:31:48.396857] E [rpc-clnt.c:365:saved_frames_unwind]
> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[ 0x7fbcadb09e8b]