|
Hi, So i have updated to 3.7.14 and i still have the same issue with NFS. based on what i have provided so far from logs and dumps do you think it's an NFS issue ? should i switch to nfs-ganesha ? the problem is, the current setup is used in a production environment, and switching the mount point of +50 VMs from native nfs to nfs-ganesha is not going to be smooth and without downtime, so i really appreciate your thoughts on this matter. -- Respectfully Mahdi A. Mahdi From: mahdi.adnan@xxxxxxxxxxx To: kdhananj@xxxxxxxxxx Date: Tue, 2 Aug 2016 08:44:16 +0300 CC: gluster-users@xxxxxxxxxxx Subject: Re: Gluster 3.7.13 NFS Crash Hi, The NFS just crashed again, latest bt; (gdb) bt #0 0x00007f0b71a9f210 in pthread_spin_lock () from /lib64/libpthread.so.0 #1 0x00007f0b72c6fcd5 in fd_anonymous (inode=0x0) at fd.c:804 #2 0x00007f0b64ca5787 in shard_common_inode_write_do (frame=0x7f0b707c062c, this=0x7f0b6002ac10) at shard.c:3716 #3 0x00007f0b64ca5a53 in shard_common_inode_write_post_lookup_shards_handler (frame=<optimized out>, this=<optimized out>) at shard.c:3769 #4 0x00007f0b64c9eff5 in shard_common_lookup_shards_cbk (frame=0x7f0b707c062c, cookie=<optimized out>, this=0x7f0b6002ac10, op_ret=0, op_errno=<optimized out>, inode=<optimized out>, buf=0x7f0b51407640, xdata=0x7f0b72f57648, postparent=0x7f0b514076b0) at shard.c:1601 #5 0x00007f0b64efe141 in dht_lookup_cbk (frame=0x7f0b7075fcdc, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, inode=0x7f0b5f1d1f58, stbuf=0x7f0b51407640, xattr=0x7f0b72f57648, postparent=0x7f0b514076b0) at dht-common.c:2174 #6 0x00007f0b651871f3 in afr_lookup_done (frame=frame@entry=0x7f0b7079a4c8, this=this@entry=0x7f0b60023ba0) at afr-common.c:1825 #7 0x00007f0b65187b84 in afr_lookup_metadata_heal_check (frame=frame@entry=0x7f0b7079a4c8, this=0x7f0b60023ba0, this@entry=0xca0bd88259f5a800) at afr-common.c:2068 #8 0x00007f0b6518834f in afr_lookup_entry_heal (frame=frame@entry=0x7f0b7079a4c8, this=0xca0bd88259f5a800, this@entry=0x7f0b60023ba0) at afr-common.c:2157 #9 0x00007f0b6518867d in afr_lookup_cbk (frame=0x7f0b7079a4c8, cookie=<optimized out>, this=0x7f0b60023ba0, op_ret=<optimized out>, op_errno=<optimized out>, inode=<optimized out>, buf=0x7f0b564e9940, xdata=0x7f0b72f708c8, postparent=0x7f0b564e99b0) at afr-common.c:2205 #10 0x00007f0b653d6e42 in client3_3_lookup_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7f0b7076354c) at client-rpc-fops.c:2981 #11 0x00007f0b72a00a30 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7f0b603393c0, pollin=pollin@entry=0x7f0b50c1c2d0) at rpc-clnt.c:764 #12 0x00007f0b72a00cef in rpc_clnt_notify (trans=<optimized out>, mydata=0x7f0b603393f0, event=<optimized out>, data="" at rpc-clnt.c:925 #13 0x00007f0b729fc7c3 in rpc_transport_notify (this=this@entry=0x7f0b60349040, event=event@entry=RPC_TRANSPORT_MSG_RECEIVED, data="">
at rpc-transport.c:546 #14 0x00007f0b678c39a4 in socket_event_poll_in (this=this@entry=0x7f0b60349040) at socket.c:2353 #15 0x00007f0b678c65e4 in socket_event_handler (fd=fd@entry=29, idx=idx@entry=17, data="" poll_in=1, poll_out=0, poll_err=0) at socket.c:2466 #16 0x00007f0b72ca0f7a in event_dispatch_epoll_handler (event=0x7f0b564e9e80, event_pool=0x7f0b7349bf20) at event-epoll.c:575 #17 event_dispatch_epoll_worker (data="" at event-epoll.c:678 #18 0x00007f0b71a9adc5 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f0b713dfced in clone () from /lib64/libc.so.6 -- Respectfully Mahdi A. Mahdi From: mahdi.adnan@xxxxxxxxxxx To: kdhananj@xxxxxxxxxx Date: Mon, 1 Aug 2016 16:31:50 +0300 CC: gluster-users@xxxxxxxxxxx Subject: Re: Gluster 3.7.13 NFS Crash Many thanks, here's the results; (gdb) p cur_block $15 = 4088 (gdb) p last_block $16 = 4088 (gdb) p local->first_block $17 = 4087 (gdb) p odirect $18 = _gf_false (gdb) p fd->flags $19 = 2 (gdb) p local->call_count $20 = 2 If you need more core dumps, i have several files i can upload. -- Respectfully Mahdi A. Mahdi From: kdhananj@xxxxxxxxxx Date: Mon, 1 Aug 2016 18:39:27 +0530 Subject: Re: Gluster 3.7.13 NFS Crash To: mahdi.adnan@xxxxxxxxxxx CC: gluster-users@xxxxxxxxxxx Sorry I didn't make myself clear. The reason I asked YOU to do it is because i tried it on my system and im not getting the backtrace (it's all question marks). At the gdb prompt, go to frame 2 by typingAttach the core to gdb. On Mon, Aug 1, 2016 at 4:55 PM, Mahdi Adnan <mahdi.adnan@xxxxxxxxxxx> wrote:
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
