On 01/04/2016 11:04 AM, Prashanth Pai wrote:
Hi,
FYI, ltrace is very helpful for libgfapi clients.
[root@hummingbird gfapi]# ltrace -c -ff -p 28906
^C% time seconds usecs/call calls function
------ ----------- ----------- --------- --------------------
43.27 5.952481 59524 100 glfs_rename
21.39 2.942866 29428 100 glfs_fsync
7.35 1.011715 10117 100 glfs_fsetxattr
7.26 0.999125 9991 100 glfs_creat
5.04 0.692812 1154 600 free
4.95 0.681092 851 800 __errno_location
4.67 0.641790 3208 200 glfs_stat
3.96 0.545340 908 600 malloc
1.26 0.173293 1732 100 glfs_close
0.84 0.115284 1152 100 glfs_write
------ ----------- ----------- --------- --------------------
100.00 13.755798 2800 total
Nice, I didn't know about this. What I am looking for with this tool is
even more granularity. i.e. per xlator information. It shouldn't be so
difficult to find information like time spent in each xlator. What fop
from fuse lead to what other fops in each of the xlator etc.
Pranith
Regards,
-Prashanth Pai
----- Original Message -----
From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
To: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>
Sent: Monday, January 4, 2016 10:49:50 AM
Subject: strace like utility for gluster
hi,
I have been thinking of coming up with a utility similar to strace
for gluster. I am sending this mail to find out if anyone else is also
thinking about it and how they are solving this problem to see if we can
come up with a concrete solution that we can implement.
These are my thoughts about the solution so far:
1) Doing trace for a brick/mount to know just what that process is
winding/unwinding (Or whatever else it wants to tell the trace process)
seems easier. We can launch a trace process which will open listener
unix socket to which the glusterfs process can send whatever it needs to
(glfstrace -h <brick-host> -p <brick-port> -s
<unix-socket-path-where-the-trace-needs-to-be-sent>). We will have to
write gf_trace() infra, which will do the job of sending the information
to this trace process only when there is a trace process trying to
listen in on what is happening inside the process.
2) Doing end-to-end tracing:
Unix socket based listening is not going to work anymore (Unless we send
the trace information in xdata or something, which doesn't seem nice to
me). We can use network sockets for sending the information from the
bricks to the trace process. So at the time of starting a trace on the
mount process, mount process will need to send an rpc to the brick
process giving the trace process hostname/port information for that
client, and bricks can send the trace information to the trace process
directly. So we can have multiple trace processes tracing different
mounts and bricks will be able to send trace information to different
trace processes.
We will need to make sure gf_trace() is not going to send the
information to trace process in the io-path.
3) Doing glfstrace -p <application-pid>. I have an approach for fuse
based mounts. I am hoping nfs/smb folks will respond if we can do
something similar in those mounts.
glfstrace process now traces the client process with extra information
i.e. frame->root->pid information in fuse-bridge which can be used to
filter only these fops executed by the application. Rest is similar to 2)
4) Doing glfstrace -c <shell-command>
glfstrace process forks, it knows the pid of child, child should wait to
hear from parent to start 'exec-of-cmd'. glfstrace process sets things
up similar to 3), once it sets up tracing, tells child to exec the cmd.
Comments are welcome :-). Happy new year by the way!!
Pranith
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
