Re: gluster IRC and use systemtap can't output glusterfs backtrace

"justglusterfs@xxxxxxxxx" <justglusterfs@xxxxxxxxx> · Wed, 14 May 2014 16:59:19 +0800

 to  Krishnan Parthasarathi:
Thank you very much for your help,  Your reply is great help to me, thank you,  
I Adopt the method of you,  modify probe_glusterfs.stap  below:

execute  command:   stap  probe_glusterfs.stap  -D MAXBACKTRACE=200 

#!/usr/bin/evn stap

probe process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").statement("*")

{

        print_ubacktrace();

}

probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so").statement("*")

{

       print_ubacktrace();

}

probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so").statement("*")

{

	print_ubacktrace();

}                       

probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/pump.so").statement("*")

{

	print_ubacktrace();

}                       

probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/replicate.so").statement("*")

{

	print_ubacktrace();

}                       

probe process("/usr/local/lib/glusterfs/3.5/xlator/protocol/client.so").statement("*")

{

	print_ubacktrace();

}

probe process("/usr/local/lib/libgfrpc.so").statement("*")

{

	print_ubacktrace();

}

probe process("/usr/local/lib/glusterfs/3.5/rpc-transport/socket.so").statement("*")

{

	print_ubacktrace();

}

probe process("/usr/local/lib/libglusterfs.so").statement("*")

{

	print_ubacktrace();

}

probe process("/usr/local/sbin/glusterfsd").statement("*")

{

	print_ubacktrace();

}

justglusterfs@xxxxxxxxx

From: Krishnan Parthasarathi
Date: 2014-05-14 14:29
To: justglusterfs
CC: gluster-devel
Subject: Re:  gluster IRC and use systemtap can't	output glusterfs backtrace
What do you mean by the back trace is not continuous?

GlusterFS code is based on asynchronous event based programming built around non-blocking
sockets. So, there is a call and a call back that execute in different threads
of execution in the mount process, for every file operation on the mount.This 
could be one of the reasons why the back trace appears discontinuous.

If you think the backtrace appears truncated then check man stap(1) for increasing
default backtrace size (See MAXBACKTRACE).

Hope that helps,
Krish

----- Original Message -----
> to  Krishnan Parthasarathi:
> I write a systemtap script（probe_glusterfs.stap）： the script content as
> follows：
> 
> probe
> process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").statement("*")
> {
> print_ubacktrace();
> }
> 
> probe
> process("/usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so").statement("*")
> {
> print_ubacktrace();
> }
> 
> probe
> process("/usr/local/lib/glusterfs/3.5/xlator/protocol/client.so").statement("*")
> {
> print_ubacktrace();
> }
> 
> when I do  "ls  /mnt/glusterfs-volume" test， I execute command:   stap
> probe_glusterfs.stap
> systemtap output as follows info:
> but the call stack information is not continuous,
> Actually, I want to watch glusterfs all function call stacks when I execute
> "ls  /mnt/glusterfs-volume"  or "mkdir  /mnt/glusterfs-volume/testdir "
> (pay attention to:  /mnt/glusterfs-volume is my glusterfs volume mount dir)
> 
> systemtap output as follows info:
>  0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a6980a0 : fuse_resolve_all+0x0/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a697b90 : fuse_resolve+0x0/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a6980a0 : fuse_resolve_all+0x0/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a698100 : fuse_resolve_all+0x60/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a69f590 : fuse_readdir_resume+0x0/0x3f0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a696660 : get_call_frame_for_req+0x0/0x270
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a69667f : get_call_frame_for_req+0x1f/0x270
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a6966d3 : get_call_frame_for_req+0x73/0x270
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a696724 : get_call_frame_for_req+0xc4/0x270
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a696769 : get_call_frame_for_req+0x109/0x270
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a6964a0 : frame_fill_groups+0x0/0x1c0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69681a : get_call_frame_for_req+0x1ba/0x270
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698106 : fuse_resolve_all+0x66/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697e36 : fuse_resolve+0x2a6/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69814e : fuse_resolve_all+0xae/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a697dde : fuse_resolve+0x24e/0x510
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb4675d2c60 : afr_readdirp+0x0/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb4675d25f0 : afr_do_readdir+0x0/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb46760bea0 : afr_local_init+0x0/0x100
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2699 : afr_do_readdir+0xa9/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb467609990 : afr_up_children_count+0x0/0x10
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760bf06 : afr_local_init+0x66/0x100
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2699 : afr_do_readdir+0xa9/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb467609960 : afr_set_elem_count_get+0x0/0x30
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760bf06 : afr_local_init+0x66/0x100
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2699 : afr_do_readdir+0xa9/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb46760b170 : afr_children_create+0x0/0x80
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d26aa : afr_do_readdir+0xba/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb467612f00 : afr_inode_get_read_ctx+0x0/0x40
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d26ca : afr_do_readdir+0xda/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb467612d70 : afr_inode_get_ctx_params+0x0/0x190
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb467612f29 : afr_inode_get_read_ctx+0x29/0x40
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d26ca : afr_do_readdir+0xda/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb467612820 : __afr_inode_ctx_get+0x0/0x110
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb467612dc7 : afr_inode_get_ctx_params+0x57/0x190
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb467612f29 : afr_inode_get_read_ctx+0x29/0x40
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d26ca : afr_do_readdir+0xda/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb46760b550 : afr_get_call_child+0x0/0x260
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d26f1 : afr_do_readdir+0x101/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb4675ec2b0 : afr_fd_ctx_get+0x0/0x60
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2704 : afr_do_readdir+0x114/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb4675ec1e0 : __afr_fd_ctx_get+0x0/0xd0
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675ec2e0 : afr_fd_ctx_get+0x30/0x60
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2704 : afr_do_readdir+0x114/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb4678389f0 : client_readdirp+0x0/0x1d0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb467843420 : client3_3_readdirp+0x0/0x4c0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467838a79 : client_readdirp+0x89/0x1d0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb46783e570 : this_fd_get_ctx+0x0/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb4678435d0 : client3_3_readdirp+0x1b0/0x4c0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467838a79 : client_readdirp+0x89/0x1d0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb46783cee0 : client_submit_request+0x0/0x4c0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467843743 : client3_3_readdirp+0x323/0x4c0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467838a79 : client_readdirp+0x89/0x1d0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb4675d2889 : afr_do_readdir+0x299/0x670
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2c73 : afr_readdirp+0x13/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4673a14b9
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
> 0x7fb46a6acf40 : fuse_graph_sync+0x0/0x170
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad2e0 : fuse_thread_proc+0x230/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46784b2b0 : client3_3_readdirp_cbk+0x0/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb4675d2ca0 : afr_readdirp_cbk+0x0/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46a6ac8c0 : fuse_readdir_cbk+0x0/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46a698f70 : send_fuse_data+0x0/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6acabd : fuse_readdir_cbk+0x1fd/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46a698da0 : send_fuse_iov+0x0/0x1d0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a698fa6 : send_fuse_data+0x36/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6acabd : fuse_readdir_cbk+0x1fd/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46a696a30 : free_fuse_state+0x0/0x150
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a696afe : free_fuse_state+0xce/0x150
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a696b0a : free_fuse_state+0xda/0x150
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46a6acac9 : fuse_readdir_cbk+0x209/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46a6acad6 : fuse_readdir_cbk+0x216/0x430
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46673f12e
> [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
> 0x7fb46760dfc0 : afr_local_cleanup+0x0/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46a699bc0 : fuse_releasedir+0x0/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46760ddd0 : afr_local_sh_cleanup+0x0/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46a6968d0 : get_fuse_state+0x0/0xf0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699bf4 : fuse_releasedir+0x34/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46760d600 : afr_xattr_array_destroy+0x0/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760de18 : afr_local_sh_cleanup+0x48/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46a696938 : get_fuse_state+0x68/0xf0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699bf4 : fuse_releasedir+0x34/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46760d5c0 : afr_reset_xattr+0x0/0x40
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760d609 : afr_xattr_array_destroy+0x9/0x20
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760de18 : afr_local_sh_cleanup+0x48/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760de33 : afr_local_sh_cleanup+0x63/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760de42 : afr_local_sh_cleanup+0x72/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfda : afr_local_cleanup+0x1a/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46760df10 : afr_local_transaction_cleanup+0x0/0xb0
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760df2f : afr_local_transaction_cleanup+0x1f/0xb0
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760df3e : afr_local_transaction_cleanup+0x2e/0xb0
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb467605120 : afr_entry_lockee_cleanup+0x0/0x70
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760df6e : afr_local_transaction_cleanup+0x5e/0xb0
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46760e0ff : afr_local_cleanup+0x13f/0x270
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46a699330 : send_fuse_err+0x0/0x140
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a696120 : fuse_ino_to_inode+0x0/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69938a : send_fuse_err+0x5a/0x140
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46783e510 : client_local_wipe+0x0/0x60
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46784b45f : client3_3_readdirp_cbk+0x1af/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46a696138 : fuse_ino_to_inode+0x18/0x40
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a69938a : send_fuse_err+0x5a/0x140
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb4678377d0 : client_releasedir+0x0/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb467847ab0 : client3_3_releasedir+0x0/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46a698da0 : send_fuse_iov+0x0/0x1d0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6993fb : send_fuse_err+0xcb/0x140
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46a696a30 : free_fuse_state+0x0/0x150
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46783e850 : this_fd_del_ctx+0x0/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b0d : client3_3_releasedir+0x5d/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a696afe : free_fuse_state+0xce/0x150
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb467847b1c : client3_3_releasedir+0x6c/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a696b0a : free_fuse_state+0xda/0x150
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb467847530 : client_fdctx_destroy+0x0/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46a6acf40 : fuse_graph_sync+0x0/0x170
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x7fb46a6ad2e0 : fuse_thread_proc+0x230/0x660
> [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
> 0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
> 0x7fb46784775f : client_fdctx_destroy+0x22f/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46784779b : client_fdctx_destroy+0x26b/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46783cee0 : client_submit_request+0x0/0x4c0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46784788f : client_fdctx_destroy+0x35f/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb4675d24e0 : afr_releasedir+0x0/0x110
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb4675d24e2 : afr_releasedir+0x2/0x110
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46760bbb0 : afr_cleanup_fd_ctx+0x0/0xe0
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb4675d259d : afr_releasedir+0xbd/0x110
> [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb4678377d0 : client_releasedir+0x0/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb467847ab0 : client3_3_releasedir+0x0/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46783e850 : this_fd_del_ctx+0x0/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b0d : client3_3_releasedir+0x5d/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb467847b1c : client3_3_releasedir+0x6c/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb467847530 : client_fdctx_destroy+0x0/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46784775f : client_fdctx_destroy+0x22f/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46784779b : client_fdctx_destroy+0x26b/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46783cee0 : client_submit_request+0x0/0x4c0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46784788f : client_fdctx_destroy+0x35f/0x4b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb467837823 : client_releasedir+0x53/0xe0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
> 0x7fb46783e050 : clnt_readdirp_rsp_cleanup+0x0/0x50
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46784b498 : client3_3_readdirp_cbk+0x1e8/0x3b0
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46784ce70 : client3_3_releasedir_cbk+0x0/0x120
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46784ce7d : client3_3_releasedir_cbk+0xd/0x120
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46784ce8b : client3_3_releasedir_cbk+0x1b/0x120
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46784ce70 : client3_3_releasedir_cbk+0x0/0x120
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46784ce7d : client3_3_releasedir_cbk+0xd/0x120
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 0x7fb46784ce8b : client3_3_releasedir_cbk+0x1b/0x120
> [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
> 0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
> 
> 
> 
> 
> justglusterfs@xxxxxxxxx
>  
> From: Krishnan Parthasarathi
> Date: 2014-05-14 11:26
> To: justglusterfs
> CC: gluster-devel
> Subject: Re: [Gluster-devel] gluster IRC and use systemtap can't output
> glusterfs backtrace
> This systemtap script limits itself to the fuse module and glusterfs binary.
> But it is possible to track all the modules that can be possibly
> (dynamically)
> loaded into a glusterfs process using system scripts.
>  
> I am not sure what kind of analysis you are looking at, across all function
> call stacks.
>  
> thanks,
> Krish
>  
> ----- Original Message -----
> > to  Krishnan Parthasarathi:
> > thank you very much for your help,
> > As you said,  I use stap -d /usr/local/sbin/glusterfs --ldd -e 'probe
> > process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").function("*"){print_ubacktrace()}'
> > ,
> > systemtap can output the fuse module function of the stack ,
> > But now I want to track glusterfs all function'call in order to analyze
> > glusterfs system, need output all function call stack ,
> > Do you have any better suggestion? or  Recommend other tools（similar to
> > systemtap）?
> >         thank you very much!
> > 
> > 
> > 
> > 
> > justglusterfs@xxxxxxxxx
> >  
> > From: Krishnan Parthasarathi
> > Date: 2014-05-13 13:38
> > To: 陈陈
> > CC: gluster-devel
> > Subject: Re: [Gluster-devel] gluster IRC and use systemtap can't output
> > glusterfs backtrace
> > To probe functions in specific translators you need to
> > specify the path to the corresponding .so file in the process
> > provider.
> >  
> > For eg, to probe functions in fuse translator,
> >  
> > stap -d /usr/local/sbin/glusterfs --ldd -e 'probe
> > process("/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so").function("*"){print_ubacktrace()}'
> >  确实可以输出fuse模块的函数栈
> > The following is the output I see, when I write files using dd on
> > a GlusterFS native mount,
> >  
> > <snip>
> >  
> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
> > 0x7f7977a1c834 : fuse_resolve+0x74/0x4f0
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]
> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
> > 0x7f7977a1e4d0 : fuse_fd_ctx_get+0x0/0x40
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1c84c : fuse_resolve+0x8c/0x4f0
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]
> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
> > 0x7f7977a1c1e0 : fuse_resolve_continue+0x0/0x100
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1c9f8 : fuse_resolve+0x238/0x4f0
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690
> > [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> > 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]
> > 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
> >  
> > </snip>
> >  
> > Hope that helps,
> > Krish
> >  
> >  
> > ----- Original Message -----
> > > See inline for response.
> > > 
> > > ----- Original Message -----
> > > > hi everyone!
> > > > I meet three question:
> > > > 
> > > > question one:
> > > > 
> > > > I use IRC client(mirc) config server is:  irc.freenode.net  or
> > > > irc.gnu.org
> > > > but always can't connect, follwing errors:
> > > > Connect retry #1 irc,freenode,net (6667) (dns pool)
> > > > unable to connect server (connect timed out)
> > > > 
> > > > 
> > > > 
> > > > question two:
> > > > I system is centos6.4，glusterfs version is 3.4.
> > > > I want to use systemtap watch glusterfs function backtrace,
> > > > when I configure glusterfs source code, execute the following
> > > > operation:
> > > > ./configure --enable--systemtap
> > > > but checking dtrace happen error，because in my system，no dtrace，my
> > > > question is --enable--systemtap why need dtrace?  it must have dtrace?
> > > 
> > > I think the systemtap folks have retained certain symbols from dtrace
> > > so that code instrumented with systemtap user specified probes are
> > > compatible
> > > with systems that have dtrace but not systemtap in them.
> > > 
> > > > 
> > > > question three:
> > > > when I do io test in glusterfs client. I execute following command:
> > > > stap -d /usr/local/sbin/glusterfs --ldd -e 'probe
> > > > process("glusterfs").function("*"){print_ubacktrace()}'
> > > > 
> > > > the result is  there is nothing output, and  the stap command S state,
> > > > no return.
> > > 
> > > For 'process' systemtap provider, you don't need to compile glusterfs
> > > with
> > > "--enable-systemtap". I am not sure why you are not seeing any of the
> > > probes
> > > firing in your stap script. Let me try this same and share my results on
> > > this
> > > thread.
> > > 
> > > Hope that helps,
> > > Krish
> > > 
> > > > 
> > > > thank you answer me!
> > > > _______________________________________________
> > > > Gluster-devel mailing list
> > > > Gluster-devel@xxxxxxxxxxx
> > > > http://supercolony.gluster.org/mailman/listinfo/gluster-devel
> > > > 
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel@xxxxxxxxxxx
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-devel
> > > 
> > 
> 

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel