Hello,
Thanks Avati for your suggestion. I tried studying the output generated my the meta xlator. One of the problems that I'm facing is that in order to enable latency measurement with meta, I need to write '1' into the the file .meta/measure_latency. But due to some reason I'm not able to do that. I don't get any permission error while writing. But the text written in the file is always the default one. So far I've only been able to change the digit by hard coding it in the meta source and building it again. I ran glusterfs under gdb and set a break point on
measure_file_write(). The redacted gdb output can be found at http://fpaste.org/104770/
As you can see when I try to write 1 (echo 1 > measure_latency) into the file, the data received in iov is "1\n" which gives the correct value in num and this->ctx->measure_latency. But somehow, measure_file_write is called again with data iov="\n\n". Therefore I feel that this write process is not doing it's job properly. If you think that I've made some mistake please do notify me.
Regards
Vipul Nayyar
As you can see when I try to write 1 (echo 1 > measure_latency) into the file, the data received in iov is "1\n" which gives the correct value in num and this->ctx->measure_latency. But somehow, measure_file_write is called again with data iov="\n\n". Therefore I feel that this write process is not doing it's job properly. If you think that I've made some mistake please do notify me.
Regards
Vipul Nayyar
On Saturday, 17 May 2014 6:24 PM, Kaushal M <kshlmster@xxxxxxxxx> wrote:
Avati that's awesome! I didn't know that the meta xlator could do that already.
On Sat, May 17, 2014 at 9:30 AM, Anand Avati <avati@xxxxxxxxxxx> wrote:
KP, Vipul,It will be awesome to get io-stats like instrumentation on the client side. Here are some further thoughts on how to implement that. If you have a recent git HEAD build, I would suggest that you explore the latency stats on the client side exposed through meta at $MNT/.meta/graphs/active/$xlator/profile. You can enable latency measurement with "echo 1 > $MNT/.meta/measure_latency". I would suggest extending these stats with the extra ones io-stats has, and make glusterfsiostats expose these stats.If you can compare libglusterfs/src/latency.c:gf_latency_begin(), gf_latency_end() and gf_latency_udpate() with the macros in io-stats.c UPDATE_PROFILE_STATS() and START_FOP_LATENCY(), you will quickly realize how a lot of logic is duplicated between io-stats and latency.c. If you can enhance latency.c and make it capture the remaining stats what io-stats is capturing, the benefits of this approach would be:- stats are already getting captured at all xlator levels, and not just at the position where io-stats is inserted- file like interface makes the stats more easily inspectable and consumable, and updated on the fly- conforms with the way rest of the internals are exposed through $MNT/.metaIn order to this, you might want to look into:- latency.c as of today captures fop count, mean latency, total time, whereas io-stats measures these along with min-time, max-time and block-size histogram.- extend gf_proc_dump_latency_info() to dump the new stats- either prettify that output like 'volume profile info' output, or JSONify it like xlators/meta/src/frames-file.c- add support for cumulative vs interval stats (store an extra copy of this->latencies[])
etc..Thanks!On Fri, Apr 25, 2014 at 9:09 PM, Krishnan Parthasarathi <kparthas@xxxxxxxxxx> wrote:
[Resending due to gluster-devel mailing list issue]
Apologies for the late reply.
glusterd uses its socket connection with brick processes (where io-stats xlator is loaded) to
gather information from io-stats via an RPC request. This facility is restricted to brick processes
as it stands today.
Some background ...
io-stats xlator is loaded, both in GlusterFS mounts and brick processes. So, we have the capabilities
to monitor I/O statistics on both sides. To collect I/O statistics at the server side, we have
# gluster volume profile VOLNAME [start | info | stop]
AND
#gluster volume top VOLNAME info [and other options]
We don't have a usable way of gathering I/O statistics (not monitoring, though the counters could be enhanced)
at the client-side, ie. for a given mount point. This is the gap glusterfsiostat aims to fill. We need to remember
that the machines hosting GlusterFS mounts may not have glusterd installed on them.
We are considering rrdtool as a possible statistics database because it seems like a natural choice for storing time-series
data. rrdtool is capable of answering high-level statistical queries on statistics that were logged in it by io-stats xlator
over and above printing running counters periodically.
Hope this gives some more clarity on what we are thinking.
thanks,
Krish
----- Original Message -----
> Probably me not understanding.
> the comment "iostats making data available to glusterd over RPC" - is what I
> latched on to. I wondered whether this meant that a socket could be opened
> that way to get at the iostats data flow.
> Cheers,
> PC
> ----- Original Message -----
> > From: "Vipul Nayyar" <nayyar_vipul@xxxxxxxxx>
>
> > To: "Paul Cuzner" <pcuzner@xxxxxxxxxx>, "Krishnan Parthasarathi"
> > <kparthas@xxxxxxxxxx>
>
> > Cc: "Vijay Bellur" <vbellur@xxxxxxxxxx>, "gluster-devel"
> > <gluster-devel@xxxxxxxxxx>
>
> > Sent: Thursday, 20 February, 2014 5:06:27 AM
>
> > Subject: Re: New project on the Forge - gstatus
>
> > Hi Paul,
>
> > I'm really not sure, if this can be done in python(at least comfortably).
> > Maybe we can tread on the same path as Justin's glusterflow in python. But
> > I
> > don't think, all the io-stats counters will be available with the way how
> > Justin's used Jeff Darcy's previous work to build his tool. I can be wrong.
> > My knowledge is a bit incomplete and based on very less experience as a
> > user
> > and an amateur Gluster developer. Please do correct me, if I can be.
>
> > Regards
>
> > Vipul Nayyar
>
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-devel
