On 05/26/2017 05:44 AM, Krutika Dhananjay wrote:
Hi,
debug/io-stats and debug/trace are immensely useful for isolating
translators that are performance bottlenecks and those that are causing
iatt inconsistencies, respectively.
There are other translators too under xlators/debug such as error-gen,
which are useful for debugging/testing our code.
The trick is to load these above and below one or more suspect
translators, run the test and analyse the output they dump and debug
your problem.
Unfortunately, there is no way to load these at specific points in the
graph using the volume-set CLI as of today. Our only option is to
manually edit the volfile and restart the process and be super-careful
not to perform *any* volume-{reset,set,profile} operation and graph
switch operations in general that could rewrite the volfile, wiping out
all previous edits to it.
I propose the following CLI for achieving the same:
# gluster volume set <VOL> {debug.trace, debug.io-stats,
debug.error-gen} <xl-name>
where <xl-name> represents the name of the translator above which you
want this translator loaded (as parent).
For example, if i have a 2x2 dis-rep volume named testvol and I want to
load trace above and below first child of DHT, I execute the following
commands:
# gluster volume set <VOL> debug.trace testvol-replicate-0
# gluster volume set <VOL> debug.trace testvol-client-0
# gluster volume set <VOL> debug.trace testvol-client-1
The corresponding debug/trace translators will be named
testvol-replicate-0-trace-parent, testvol-client-0-trace-parent,
testvol-client-1-trace-parent and so on.
To revert the change, the user simply uses volume-reset CLI:
# gluster volume reset <VOL> testvol-replicate-0-trace-parent
# gluster volume reset <VOL> testvol-client-0-trace-parent
# gluster volume reset <VOL> testvol-client-1-trace-parent
What should happen when the translator with a trace/io-stat/error-gen
parent gets disabled?
Well glusterd should be made to take care to remove the trace xl too
from the graph.
Comments and suggestions welcome.
+1, dynamic placement of io-stats was something that I added to this
spec [1] as well. So I am all for the change.
I have one problem though that bothered me when I wrote the spec,
currently brick vol files are static, and do not undergo a graph change
(or code is not yet ready to do that). So when we want to do this on the
bricks, what happens? Do you have solutions for the same? I am
interested, hence asking!
[1] Initial feature description for improved io-stats:
https://review.gluster.org/#/c/16558/1/under_review/Performance_monitoring_and_debugging.md
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel
