Re: Query specific to getting crash

Mohit Agrawal <moagrawa@xxxxxxxxxx> · Mon, 9 Oct 2017 14:07:23 +0530

+

On Mon, Oct 9, 2017 at 11:33 AM, Mohit Agrawal <moagrawa@xxxxxxxxxx> wrote:

On Mon, Oct 9, 2017 at 11:16 AM, Mohit Agrawal <moagrawa@xxxxxxxxxx> wrote:
Hi All,

For specific to this patch(https://review.gluster.org/#/c/18436/) i am getting crash in nfs(only once) for the 
test case (./tests/basic/mount-nfs-auth.t), although i tried to execute the same test case in a loop on centos
machine but i have not found any crash.

After anaylys the crash it seems cache(entry) is invalidate in thread 10 and same it is trying to access
in thread 1. 

>>>>>>>>>>>>>>>>>>>.

(gdb) thread 1
[Switching to thread 1 (Thread 0x7fe852cfe700 (LWP 19073))]#0  0x00007fe859665c85 in auth_cache_lookup (
    cache=0x7fe854027db0, fh=0x7fe84466684c, host_addr=0x7fe844565e40 "23.253.175.80", 
    timestamp=0x7fe852cfb1e0, can_write=0x7fe852cfb1dc)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/auth-cache.c:295
295	                *can_write = lookup_res->item->opts->rw;
(gdb) bt
#0  0x00007fe859665c85 in auth_cache_lookup (cache=0x7fe854027db0, fh=0x7fe84466684c, 
    host_addr=0x7fe844565e40 "23.253.175.80", timestamp=0x7fe852cfb1e0, can_write=0x7fe852cfb1dc)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/auth-cache.c:295
#1  0x00007fe859665ebc in is_nfs_fh_cached (cache=0x7fe854027db0, fh=0x7fe84466684c, 
    host_addr=0x7fe844565e40 "23.253.175.80")
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/auth-cache.c:390
#2  0x00007fe85962b82c in mnt3_check_cached_fh (ms=0x7fe854023d60, fh=0x7fe84466684c, 
    host_addr=0x7fe844565e40 "23.253.175.80", is_write_op=_gf_false)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/mount3.c:1954
#3  0x00007fe85962ba92 in _mnt3_authenticate_req (ms=0x7fe854023d60, req=0x7fe844679148, 
    fh=0x7fe84466684c, path=0x0, authorized_export=0x0, authorized_host=0x0, is_write_op=_gf_false)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/mount3.c:2011
#4  0x00007fe85962bf65 in mnt3_authenticate_request (ms=0x7fe854023d60, req=0x7fe844679148, 
    fh=0x7fe84466684c, volname=0x0, path=0x0, authorized_path=0x0, authorized_host=0x0, 
    is_write_op=_gf_false)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/mount3.c:2130
#5  0x00007fe859652370 in nfs3_fh_auth_nfsop (cs=0x7fe8446663c8, is_write_op=_gf_false)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3-helpers.c:3981
#6  0x00007fe85963631a in nfs3_lookup_resume (carg=0x7fe8446663c8)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3.c:155---Type <return> to continue, or q <return> to quit---
9
#7  0x00007fe859651b98 in nfs3_fh_resolve_entry_hard (cs=0x7fe8446663c8)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3-helpers.c:3791
#8  0x00007fe859651e35 in nfs3_fh_resolve_entry (cs=0x7fe8446663c8)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3-helpers.c:3844
#9  0x00007fe859651e94 in nfs3_fh_resolve_resume (cs=0x7fe8446663c8)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3-helpers.c:3862
#10 0x00007fe8596520ad in nfs3_fh_resolve_root (cs=0x7fe8446663c8)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3-helpers.c:3915
#11 0x00007fe85965245f in nfs3_fh_resolve_and_resume (cs=0x7fe8446663c8, fh=0x7fe852cfc980, 
    entry=0x7fe852cfc9c0 "test-bg-write", resum_fn=0x7fe85963621d <nfs3_lookup_resume>)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3-helpers.c:4011
#12 0x00007fe859636dcf in nfs3_lookup (req=0x7fe844679148, fh=0x7fe852cfc980, fhlen=52, 
    name=0x7fe852cfc9c0 "test-bg-write")
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3.c:1620
#13 0x00007fe85963703f in nfs3svc_lookup (req=0x7fe844679148)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/nfs3.c:1666
#14 0x00007fe86765f585 in rpcsvc_handle_rpc_call (svc=0x7fe854022a00, trans=0x7fe8545c1fa0, 
    msg=0x7fe844334610)
---Type <return> to continue, or q <return> to quit---
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/rpc/rpc-lib/src/rpcsvc.c:711
#15 0x00007fe86765f8f8 in rpcsvc_notify (trans=0x7fe8545c1fa0, mydata=0x7fe854022a00, 
    event=RPC_TRANSPORT_MSG_RECEIVED, data="">    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/rpc/rpc-lib/src/rpcsvc.c:805
#16 0x00007fe867665458 in rpc_transport_notify (this=0x7fe8545c1fa0, event=RPC_TRANSPORT_MSG_RECEIVED, 
    data="">    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/rpc/rpc-lib/src/rpc-transport.c:538
#17 0x00007fe85c44561e in socket_event_poll_in (this=0x7fe8545c1fa0, notify_handled=_gf_true)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/rpc/rpc-transport/socket/src/socket.c:2319
#18 0x00007fe85c445cb1 in socket_event_handler (fd=12, idx=8, gen=103, data="" poll_in=1, 
    poll_out=0, poll_err=0)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/rpc/rpc-transport/socket/src/socket.c:2475
#19 0x00007fe867917fd7 in event_dispatch_epoll_handler (event_pool=0x7030d0, event=0x7fe852cfde70)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/libglusterfs/src/event-epoll.c:583
#20 0x00007fe8679182d9 in event_dispatch_epoll_worker (data="">    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/libglusterfs/src/event-epoll.c:659
#21 0x00007fe866b7baa1 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fe8664e3bcd in clone () from /lib64/libc.so.6

(gdb) thread 10
[Switching to thread 10 (Thread 0x7fe858ed2700 (LWP 19051))]#0  0x00007fe866b82334 in __lll_lock_wait ()
   from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fe866b82334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fe866b7d5d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fe866b7d4a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fe8678a9844 in _gf_msg (
    domain=0x7fe85966a448 "ot/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/mount3.c", file=0x7fe85966a3f8 "/lib/glusterd/nfs/exports", function=0x7fe85966b5e0 "init", line=3878, 
    level=GF_LOG_INFO, errnum=0, trace=0, msgid=112151, fmt=0x7fe85966b3b4 "")
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/libglusterfs/src/logging.c:2081
#4  0x00007fe859630287 in _mnt3_auth_param_refresh_thread (argv=0x7fe854023d60)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/mount3.c:3877
#5  0x00007fe866b7baa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe8664e3bcd in clone () from /lib64/libc.so.6
(gdb) p mstate
No symbol "mstate" in current context.
(gdb) f 4
#4  0x00007fe859630287 in _mnt3_auth_param_refresh_thread (argv=0x7fe854023d60)
    at /home/jenkins/root/workspace/my_glusterfs_build/glusterfs-4.0dev/xlators/nfs/server/src/mount3.c:3877
3877

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>.

In first level analysis i don't think it is related to my patch, please check and response on the same if you have seen earlier also.

For specific to core you can access core from this link https://build.gluster.org/job/centos6-regression/6759/console

Regards
Mohit Agrawal

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel