|
Got access to the slave machine. Had a
look at the BT. The test-case that keeps failing spuriously is
kill_glusterd in cluster.rc, and the bt looks like the following.
Looks like the volinfo is corrupted. Is looks like it was trying
to access peer data after cleanup_and _exit was called.
[2015-07-19 10:23:31.799681] W [glusterfsd.c:1214:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x79d1) [0x7f3f744729d1] -->glusterd(glusterfs_sigwaiter+0xe4) [0x409734] -->glusterd(cleanup_and_exit+0x87) [0x407ba7] ) 0-: received signum (15), shutting down [2015-07-19 10:23:31.802507] W [socket.c:637:__socket_rwv] 0-management: readv on 127.1.1.3:24007 failed (No data available) [2015-07-19 10:23:31.802581] I [timer.c:43:gf_timer_call_after] (-->/build/install/lib/libgfrpc.so.0(rpc_transport_notify+0x12f) [0x7f3f74f57aeb] -->/build/install/lib/libgfrpc.so.0(rpc_clnt_notify+0x1ca) [0x7f3f74f5b644] -->/build/install/lib/libglusterfs.so.0(gf_timer_call_after+0xfb) [0x7f3f751b9517] ) 0-timer: ctx cleanup started [2015-07-19 10:23:31.802608] I [MSGID: 106004] [glusterd-handler.c:5047:__glusterd_peer_rpc_notify] 0-management: Peer <127.1.1.3> (<377dd370-1401-452b-831e-cbaf4340a376>), in state <Peer in Cluster>, has disconnected from glusterd. [2015-07-19 10:23:31.804501] W [socket.c:637:__socket_rwv] 0-management: readv on 127.1.1.1:24007 failed (No data available) [2015-07-19 10:23:31.804567] I [timer.c:43:gf_timer_call_after] (-->/build/install/lib/libgfrpc.so.0(rpc_transport_notify+0x12f) [0x7f3f74f57aeb] -->/build/install/lib/libgfrpc.so.0(rpc_clnt_notify+0x1ca) [0x7f3f74f5b644] -->/build/install/lib/libglusterfs.so.0(gf_timer_call_after+0xfb) [0x7f3f751b9517] ) 0-timer: ctx cleanup started [2015-07-19 10:23:31.804612] I [MSGID: 106004] [glusterd-handler.c:5047:__glusterd_peer_rpc_notify] 0-management: Peer <127.1.1.1> (<fd3a8e54-a8fc-47d4-93fa-ed1a9e2c78fc>), in state <Peer in Cluster>, has disconnected from glusterd. #0 0x00007f8f3c400e2c in vfprintf () from ./lib64/libc.so.6 #1 0x00007f8f3c428752 in vsnprintf () from ./lib64/libc.so.6 #2 0x00007f8f3c408223 in snprintf () from ./lib64/libc.so.6 #3 0x00007f8f32cded8d in glusterd_volume_stop_glusterfs (volinfo=0x253ee30, brickinfo=0x2549f60, del_brick=_gf_false) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-utils.c:1754 #4 0x00007f8f32cebd18 in glusterd_brick_stop (volinfo=0x253ee30, brickinfo=0x2549f60, del_brick=_gf_false) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-utils.c:5458 #5 0x00007f8f32d76804 in glusterd_snap_volume_remove (rsp_dict=0x7f8f1800102c, snap_vol=0x253ee30, remove_lvm=_gf_false, force=_gf_false) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot.c:2897 #6 0x00007f8f32d76daf in glusterd_snap_remove (rsp_dict=0x7f8f1800102c, snap=0x25388a0, remove_lvm=_gf_false, force=_gf_false) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot.c:3005 #7 0x00007f8f32da2e58 in glusterd_compare_and_update_snap (peer_data=0x7f8f180024fc, snap_count=1, peername=0x7f8f18002370 "127.1.1.1", peerid=0x7f8f180023e0 "\375:\216T\250\374Gԓ\372\355\032\236,x\374p#") at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot-utils.c:1849 #8 0x00007f8f32da311f in glusterd_compare_friend_snapshots (peer_data=0x7f8f180024fc, peername=0x7f8f18002370 "127.1.1.1", peerid=0x7f8f180023e0 "\375:\216T\250\374Gԓ\372\355\032\236,x\374p#") at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot-utils.c:1904 #9 0x00007f8f32cc49b3 in glusterd_ac_handle_friend_add_req (event=0x7f8f180023d0, ctx=0x7f8f18002460) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-sm.c:831 #10 0x00007f8f32cc5250 in glusterd_friend_sm () at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-sm.c:1253 #11 0x00007f8f32cbadd4 in __glusterd_handle_incoming_friend_req (req=0x7f8f1800128c) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:2541 #12 0x00007f8f32cb36aa in glusterd_big_locked_handler (req=0x7f8f1800128c, actor_fn=0x7f8f32cbac38 <__glusterd_handle_incoming_friend_req>) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:79 #13 0x00007f8f32cbae0a in glusterd_handle_incoming_friend_req (req=0x7f8f1800128c) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:2551 #14 0x00007f8f3d61706d in rpcsvc_handle_rpc_call (svc=0x24b8430, trans=0x7f8f20008e20, msg=0x7f8f18000e20) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpcsvc.c:699 #15 0x00007f8f3d6173e0 in rpcsvc_notify (trans=0x7f8f20008e20, mydata=0x24b8430, event=RPC_TRANSPORT_MSG_RECEIVED, data=""> at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpcsvc.c:793 #16 0x00007f8f3d61caeb in rpc_transport_notify (this=0x7f8f20008e20, event=RPC_TRANSPORT_MSG_RECEIVED, data=""> at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:538 #17 0x00007f8f3134187b in socket_event_poll_in (this=0x7f8f20008e20) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2285 #18 0x00007f8f31341dd1 in socket_event_handler (fd=20, idx=10, data="" poll_in=1, poll_out=0, poll_err=0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2398 #19 0x00007f8f3d8d09e4 in event_dispatch_epoll_handler (event_pool=0x249ec90, event=0x7f8f2dfb5e70) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:570 #20 0x00007f8f3d8d0dd2 in event_dispatch_epoll_worker (data="" at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:673 #21 0x00007f8f3cb379d1 in start_thread () from ./lib64/libpthread.so.0 #22 0x00007f8f3c4a18fd in clone () from ./lib64/libc.so.6 Regards, Avra On 07/20/2015 04:38 PM, Avra Sengupta wrote:
|
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
