On 31/07/15 06:27, Stijn De Weirdt wrote:
wouldn't it be nice that ceph does something like this in background
(some sort of network-scrub). debugging network like this is not that
easy (can't expect admins to install e.g. perfsonar on all nodes
and/or clients)
something like: every X min, each service X pick a service Y on
another host (assuming X and Y will exchange some communication at
some point; like osd with other osd), send 1MB of data, and make the
timing data available so we can monitor it and detect underperforming
links over time.
ideally clients also do this, but not sure where they should
report/store the data.
interpreting the data can be a bit tricky, but extreme outliers will
be spotted easily, and the main issue with this sort of debugging is
collecting the data.
simply reporting / keeping track of ongoing communications is already
a big step forward, but then we need to have the size of the exchanged
data to allow interpretation (and the timing should be about the
network part, not e.g. flush data to disk in case of an osd). (and
obviously sampling is enough, no need to have details of every bit send).
Yes, it's a reasonable concept, although it's not clear that we'd
necessarily want it built into existing ceph services. For example,
where there are several OSDs running on a host, we don't really want all
the OSDs redundantly verifying that particular host's network
functionality. This use case is a pretty good argument for a ceph
supervisor service of some kind that exists on a one-per-host basis.
The trick is finding someone with time to write it :-)
The prior art here is Lustres LNET self test (LST) which exists for
exactly these reasons (Mark will have memories of this too I'm sure).
John
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
