Re: strace like utility for gluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 01/04/2016 11:04 AM, Prashanth Pai wrote:
Hi,

FYI, ltrace is very helpful for libgfapi clients.

[root@hummingbird gfapi]# ltrace -c -ff -p 28906

^C% time     seconds  usecs/call     calls      function
------ ----------- ----------- --------- --------------------
  43.27    5.952481       59524       100 glfs_rename
  21.39    2.942866       29428       100 glfs_fsync
   7.35    1.011715       10117       100 glfs_fsetxattr
   7.26    0.999125        9991       100 glfs_creat
   5.04    0.692812        1154       600 free
   4.95    0.681092         851       800 __errno_location
   4.67    0.641790        3208       200 glfs_stat
   3.96    0.545340         908       600 malloc
   1.26    0.173293        1732       100 glfs_close
   0.84    0.115284        1152       100 glfs_write
------ ----------- ----------- --------- --------------------
100.00   13.755798                  2800 total
Nice, I didn't know about this. What I am looking for with this tool is even more granularity. i.e. per xlator information. It shouldn't be so difficult to find information like time spent in each xlator. What fop from fuse lead to what other fops in each of the xlator etc.

Pranith

Regards,
  -Prashanth Pai

----- Original Message -----
From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
To: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>
Sent: Monday, January 4, 2016 10:49:50 AM
Subject:  strace like utility for gluster

hi,
       I have been thinking of coming up with a utility similar to strace
for gluster. I am sending this mail to find out if anyone else is also
thinking about it and how they are solving this problem to see if we can
come up with a concrete solution that we can implement.

These are my thoughts about the solution so far:

1) Doing trace for a brick/mount to know just what that process is
winding/unwinding (Or whatever else it wants to tell the trace process)
seems easier. We can launch a trace process which will open listener
unix socket to which the glusterfs process can send whatever it needs to
(glfstrace -h <brick-host> -p <brick-port> -s
<unix-socket-path-where-the-trace-needs-to-be-sent>). We will have to
write gf_trace() infra, which will do the job of sending the information
to this trace process only when there is a trace process trying to
listen in on what is happening inside the process.

2) Doing end-to-end tracing:
Unix socket based listening is not going to work anymore (Unless we send
the trace information in xdata or something, which doesn't seem nice to
me). We can use network sockets for sending the information from the
bricks to the trace process. So at the time of starting a trace on the
mount process, mount process will need to send an rpc to the brick
process giving the trace process hostname/port information for that
client, and bricks can send the trace information to the trace process
directly. So we can have multiple trace processes tracing different
mounts and bricks will be able to send trace information to different
trace processes.

We will need to make sure gf_trace() is not going to send the
information to trace process in the io-path.

3) Doing glfstrace -p <application-pid>. I have an approach for fuse
based mounts. I am hoping nfs/smb folks will respond if we can do
something similar in those mounts.

glfstrace process now traces the client process with extra information
i.e. frame->root->pid information in fuse-bridge which can be used to
filter only these fops executed by the application. Rest is similar to 2)

4) Doing glfstrace -c <shell-command>

glfstrace process forks, it knows the pid of child, child should wait to
hear from parent to start 'exec-of-cmd'. glfstrace process sets things
up similar to 3), once it sets up tracing, tells child to exec the cmd.

Comments are welcome :-). Happy new year by the way!!

Pranith
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel



[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux