Got access to the slave machine. Had a look at the BT. The test-case
that keeps failing spuriously is kill_glusterd in cluster.rc, and the bt
looks like the following. Looks like the volinfo is corrupted. Is looks
like it was trying to access peer data after cleanup_and _exit was called.
[2015-07-19 10:23:31.799681] W [glusterfsd.c:1214:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x79d1) [0x7f3f744729d1]
-->glusterd(glusterfs_sigwaiter+0xe4) [0x409734]
-->glusterd(cleanup_and_exit+0x87) [0x407ba7] ) 0-: received signum
(15), shutting down
[2015-07-19 10:23:31.802507] W [socket.c:637:__socket_rwv] 0-management:
readv on 127.1.1.3:24007 failed (No data available)
[2015-07-19 10:23:31.802581] I [timer.c:43:gf_timer_call_after]
(-->/build/install/lib/libgfrpc.so.0(rpc_transport_notify+0x12f)
[0x7f3f74f57aeb]
-->/build/install/lib/libgfrpc.so.0(rpc_clnt_notify+0x1ca)
[0x7f3f74f5b644]
-->/build/install/lib/libglusterfs.so.0(gf_timer_call_after+0xfb)
[0x7f3f751b9517] ) 0-timer: ctx cleanup started
[2015-07-19 10:23:31.802608] I [MSGID: 106004]
[glusterd-handler.c:5047:__glusterd_peer_rpc_notify] 0-management: Peer
<127.1.1.3> (<377dd370-1401-452b-831e-cbaf4340a376>), in state <Peer in
Cluster>, has disconnected from glusterd.
[2015-07-19 10:23:31.804501] W [socket.c:637:__socket_rwv] 0-management:
readv on 127.1.1.1:24007 failed (No data available)
[2015-07-19 10:23:31.804567] I [timer.c:43:gf_timer_call_after]
(-->/build/install/lib/libgfrpc.so.0(rpc_transport_notify+0x12f)
[0x7f3f74f57aeb]
-->/build/install/lib/libgfrpc.so.0(rpc_clnt_notify+0x1ca)
[0x7f3f74f5b644]
-->/build/install/lib/libglusterfs.so.0(gf_timer_call_after+0xfb)
[0x7f3f751b9517] ) 0-timer: ctx cleanup started
[2015-07-19 10:23:31.804612] I [MSGID: 106004]
[glusterd-handler.c:5047:__glusterd_peer_rpc_notify] 0-management: Peer
<127.1.1.1> (<fd3a8e54-a8fc-47d4-93fa-ed1a9e2c78fc>), in state <Peer in
Cluster>, has disconnected from glusterd.
#0 0x00007f8f3c400e2c in vfprintf () from ./lib64/libc.so.6
#1 0x00007f8f3c428752 in vsnprintf () from ./lib64/libc.so.6
#2 0x00007f8f3c408223 in snprintf () from ./lib64/libc.so.6
#3 0x00007f8f32cded8d in glusterd_volume_stop_glusterfs
(volinfo=0x253ee30, brickinfo=0x2549f60, del_brick=_gf_false)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-utils.c:1754
#4 0x00007f8f32cebd18 in glusterd_brick_stop (volinfo=0x253ee30,
brickinfo=0x2549f60, del_brick=_gf_false)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-utils.c:5458
#5 0x00007f8f32d76804 in glusterd_snap_volume_remove
(rsp_dict=0x7f8f1800102c, snap_vol=0x253ee30, remove_lvm=_gf_false,
force=_gf_false)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot.c:2897
#6 0x00007f8f32d76daf in glusterd_snap_remove (rsp_dict=0x7f8f1800102c,
snap=0x25388a0, remove_lvm=_gf_false, force=_gf_false)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot.c:3005
#7 0x00007f8f32da2e58 in glusterd_compare_and_update_snap
(peer_data=0x7f8f180024fc, snap_count=1, peername=0x7f8f18002370
"127.1.1.1", peerid=0x7f8f180023e0
"\375:\216T\250\374Gԓ\372\355\032\236,x\374p#")
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot-utils.c:1849
#8 0x00007f8f32da311f in glusterd_compare_friend_snapshots
(peer_data=0x7f8f180024fc, peername=0x7f8f18002370 "127.1.1.1",
peerid=0x7f8f180023e0 "\375:\216T\250\374Gԓ\372\355\032\236,x\374p#")
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-snapshot-utils.c:1904
#9 0x00007f8f32cc49b3 in glusterd_ac_handle_friend_add_req
(event=0x7f8f180023d0, ctx=0x7f8f18002460)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-sm.c:831
#10 0x00007f8f32cc5250 in glusterd_friend_sm () at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-sm.c:1253
#11 0x00007f8f32cbadd4 in __glusterd_handle_incoming_friend_req
(req=0x7f8f1800128c) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:2541
#12 0x00007f8f32cb36aa in glusterd_big_locked_handler
(req=0x7f8f1800128c, actor_fn=0x7f8f32cbac38
<__glusterd_handle_incoming_friend_req>)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:79
#13 0x00007f8f32cbae0a in glusterd_handle_incoming_friend_req
(req=0x7f8f1800128c) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:2551
#14 0x00007f8f3d61706d in rpcsvc_handle_rpc_call (svc=0x24b8430,
trans=0x7f8f20008e20, msg=0x7f8f18000e20) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpcsvc.c:699
#15 0x00007f8f3d6173e0 in rpcsvc_notify (trans=0x7f8f20008e20,
mydata=0x24b8430, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f8f18000e20)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpcsvc.c:793
#16 0x00007f8f3d61caeb in rpc_transport_notify (this=0x7f8f20008e20,
event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f8f18000e20)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:538
#17 0x00007f8f3134187b in socket_event_poll_in (this=0x7f8f20008e20) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2285
#18 0x00007f8f31341dd1 in socket_event_handler (fd=20, idx=10,
data=0x7f8f20008e20, poll_in=1, poll_out=0, poll_err=0)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2398
#19 0x00007f8f3d8d09e4 in event_dispatch_epoll_handler
(event_pool=0x249ec90, event=0x7f8f2dfb5e70) at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:570
#20 0x00007f8f3d8d0dd2 in event_dispatch_epoll_worker (data=0x24a97f0)
at
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:673
#21 0x00007f8f3cb379d1 in start_thread () from ./lib64/libpthread.so.0
#22 0x00007f8f3c4a18fd in clone () from ./lib64/libc.so.6
