----- Original Message ----- > From: "Raghavendra Gowdappa" <rgowdapp@xxxxxxxxxx> > To: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>, "Nithya Balachandran" <nbalacha@xxxxxxxxxx> > Sent: Wednesday, February 10, 2016 10:16:45 AM > Subject: Re: crash in dht in https://build.gluster.org/job/rackspace-regression-2GB-triggered/18134/consoleFull > > Seems like a double unwind (or frame corruption in tier) > (gdb) down > #5 0x00007f3f8c34c0ac in dht_lookup_dir_cbk (frame=0x7f3f6c002a0c, > cookie=0x7f3f6c0064bc, this=0x7f3f8803bad0, op_ret=-1, op_errno=2, > inode=0x0, > stbuf=0x7f3f6c0069d4, xattr=0x0, postparent=0x7f3f6c006c04) > at > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:724 > 724 ret = dht_selfheal_directory (frame, dht_lookup_selfheal_cbk, > (gdb) p *frame > $13 = {root = 0x29f00000000000, parent = 0xadc0de00007f3f6c, frames = {next = > 0x50e4000000000de, prev = 0x3000007f3f88}, local = 0xac0000000000, > this = 0x29f00000000000, ret = 0xadc0de00007f3f6c, ref_count = 222, lock = > 84819968, cookie = 0x3000007f3f88, complete = _gf_false, op = 44032, begin > = { > tv_sec = 11804356835803136, tv_usec = -5926493018029867156}, end = > {tv_sec = 364298988607766750, tv_usec = 52776566472584}, > wind_from = 0xac0000000000 <Address 0xac0000000000 out of bounds>, wind_to > = 0x29f00000000000 <Address 0x29f00000000000 out of bounds>, > unwind_from = 0xffffff00007f3f6c <Address 0xffffff00007f3f6c out of > bounds>, unwind_to = 0xffffffffffffffff <Address 0xffffffffffffffff out of > bounds>} > (gdb) p this->name > $14 = 0x7f3f8803b640 "patchy-tier-dht" (gdb) up #6 0x00007f3f8fb5ea98 in dht_lookup_dir_cbk (frame=0x7f3f6c0064bc, cookie=0x7f3f6c0084dc, this=0x7f3f8803aa80, op_ret=-1, op_errno=2, inode=0x7f3f6c0013ec, stbuf=0x7f3f94bc2860, xattr=0x0, postparent=0x7f3f94bc27f0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:714 714 DHT_STACK_UNWIND (lookup, frame, local->op_ret, local->op_errno, (gdb) p this->name $22 = 0x7f3f88039510 "patchy-hot-dht" (gdb) p frame->parent $23 = (call_frame_t *) 0xadc0de00007f3f6c (gdb) p *frame->parent Cannot access memory at address 0xadc0de00007f3f6c (gdb) p *frame $24 = {root = 0x64a00000000000, parent = 0xadc0de00007f3f6c, frames = {next = 0x50e4000000000de, prev = 0x3000007f3f88}, local = 0xac0000000000, this = 0x64a00000000000, ret = 0xadc0de00007f3f6c, ref_count = 222, lock = 84819968, cookie = 0x3000007f3f88, complete = _gf_false, op = 44032, begin = { tv_sec = 28323419531509760, tv_usec = -5926493018029867156}, end = {tv_sec = 364298988607766750, tv_usec = 52776566472584}, wind_from = 0xac0000000000 <Address 0xac0000000000 out of bounds>, wind_to = 0x64a00000000000 <Address 0x64a00000000000 out of bounds>, unwind_from = 0xffffff00007f3f6c <Address 0xffffff00007f3f6c out of bounds>, unwind_to = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>} (gdb) p this->name $25 = 0x7f3f88039510 "patchy-hot-dht" (gdb) up #7 0x00007f3f8fddb23a in client3_3_lookup_cbk (req=0x7f3f6c0090ac, iov=0x7f3f6c0090ec, count=1, myframe=0x7f3f6c0084dc) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c:3028 3028 /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c: No such file or directory. (gdb) p frame $26 = (call_frame_t *) 0x7f3f6c0084dc (gdb) p *frame $27 = {root = 0x84c00000000000, parent = 0xadc0de00007f3f6c, frames = {next = 0x54d2000000000de, prev = 0x3000007f3f88}, local = 0xac0000000000, this = 0x84c00000000000, ret = 0xadc0de00007f3f6c, ref_count = 222, lock = 88940544, cookie = 0x3000007f3f88, complete = _gf_false, op = 44032, begin = { tv_sec = 37365803158339584, tv_usec = -5926493018029867156}, end = {tv_sec = 381996727768449246, tv_usec = 52776566472584}, wind_from = 0xac0000000000 <Address 0xac0000000000 out of bounds>, wind_to = 0x84c00000000000 <Address 0x84c00000000000 out of bounds>, unwind_from = 0xffffff00007f3f6c <Address 0xffffff00007f3f6c out of bounds>, unwind_to = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>} (gdb) p this->name $28 = 0x7f3f880202d0 "patchy-client-5" Seems like entire stack is corrupted > No, I am not aware of this crash. > > ----- Original Message ----- > > From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> > > To: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>, "Raghavendra Gowdappa" > > <rgowdapp@xxxxxxxxxx>, "Nithya Balachandran" > > <nbalacha@xxxxxxxxxx> > > Sent: Wednesday, February 10, 2016 5:51:32 AM > > Subject: crash in dht in > > https://build.gluster.org/job/rackspace-regression-2GB-triggered/18134/consoleFull > > > > hi, > > I see the following crash. Is this a known issue? > > (gdb) bt > > #0 0x00007f3f8c339fb4 in dht_selfheal_dir_setattr > > (frame=0x7f3f6c002a0c, loc=0x7f3f6c000944, stbuf=0x7f3f6c0009d4, > > valid=16777215, > > layout=0x7f3f6c004140) at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1087 > > #1 0x00007f3f8c33a4f8 in dht_selfheal_dir_mkdir_cbk > > (frame=0x7f3f6c002a0c, cookie=0x7f3f9004201c, this=0x7f3f8803bad0, > > op_ret=-1, op_errno=5, > > inode=0x0, stbuf=0x0, preparent=0x0, postparent=0x0, xdata=0x0) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1147 > > #2 0x00007f3f8fb81f31 in dht_mkdir (frame=0x7f3f9004201c, > > this=0x7f3f8803aa80, loc=0x7f3f6c000944, mode=16877, umask=0, > > params=0x7f3f9003594c) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:6710 > > #3 0x00007f3f8c33ad1f in dht_selfheal_dir_mkdir (frame=0x7f3f6c002a0c, > > loc=0x7f3f6c000944, layout=0x7f3f6c004140, force=0) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1256 > > #4 0x00007f3f8c33c4c8 in dht_selfheal_directory (frame=0x7f3f6c002a0c, > > dir_cbk=0x7f3f8c349588 <dht_lookup_selfheal_cbk>, loc=0x7f3f6c000944, > > layout=0x7f3f6c004140) at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1837 > > #5 0x00007f3f8c34c0ac in dht_lookup_dir_cbk (frame=0x7f3f6c002a0c, > > cookie=0x7f3f6c0064bc, this=0x7f3f8803bad0, op_ret=-1, op_errno=2, > > inode=0x0, > > stbuf=0x7f3f6c0069d4, xattr=0x0, postparent=0x7f3f6c006c04) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:724 > > #6 0x00007f3f8fb5ea98 in dht_lookup_dir_cbk (frame=0x7f3f6c0064bc, > > cookie=0x7f3f6c0084dc, this=0x7f3f8803aa80, op_ret=-1, op_errno=2, > > inode=0x7f3f6c0013ec, stbuf=0x7f3f94bc2860, xattr=0x0, > > postparent=0x7f3f94bc27f0) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:714 > > #7 0x00007f3f8fddb23a in client3_3_lookup_cbk (req=0x7f3f6c0090ac, > > iov=0x7f3f6c0090ec, count=1, myframe=0x7f3f6c0084dc) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c:3028 > > #8 0x00007f3fa0c2d42a in rpc_clnt_handle_reply (clnt=0x7f3f8806cdb0, > > pollin=0x7f3f90041c90) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:759 > > #9 0x00007f3fa0c2d8c8 in rpc_clnt_notify (trans=0x7f3f8806d240, > > mydata=0x7f3f8806cde0, event=RPC_TRANSPORT_MSG_RECEIVED, > > data=0x7f3f90041c90) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:900 > > #10 0x00007f3fa0c29b5a in rpc_transport_notify (this=0x7f3f8806d240, > > event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f3f90041c90) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:541 > > #11 0x00007f3f961eadcb in socket_event_poll_in (this=0x7f3f8806d240) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2231 > > #12 0x00007f3f961eb321 in socket_event_handler (fd=18, idx=12, > > data=0x7f3f8806d240, poll_in=1, poll_out=0, poll_err=0) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2344 > > #13 0x00007f3fa0ec61a8 in event_dispatch_epoll_handler > > (event_pool=0x121fce0, event=0x7f3f94bc2e70) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:571 > > #14 0x00007f3fa0ec6596 in event_dispatch_epoll_worker (data=0x125e1a0) > > at > > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:674 > > #15 0x00007f3fa0144a51 in start_thread () from ./lib64/libpthread.so.0 > > #16 0x00007f3f9faae93d in clone () from ./lib64/libc.so.6 > > > > > > Pranith > > > _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
