Hi all,
[this is a re-send of a mail from yesterday that didn't make it,
probably due to an attachment]
It has always annoyed me that we don't provide a simple progress bar
indicator for things like the migration of data from an OSD when it's
marked out, the rebalance that happens when we add a new OSD, or
scrubbing the PGs on an OSD.
I've experimented a bit with adding user-visible progress bars for some
of the simple cases (screenshot at http://imgur.com/OaifxMf). The code
is here:
https://github.com/ceph/ceph/blob/wip-progress-events/src/mon/ProgressEvent.cc
This is based on a series of "ProgressEvent" classes that are
instantiated when certain things happen, like marking and OSD in or
out. They provide an init() hook that captures whatever state is needed
at the start of the operation (generally noting which PGs are affected)
and a tick() hook that checks whether the affected PGs have reached
their final state.
Clearly, while this is simple for the simple cases, there are lots of
instances where things will overlap: a PG can get moved again while it's
being backfilled following a particular OSD going out. These progress
indicators don't have to capture that complexity, but the goal would be
to make sure they did complete eventually rather than getting
stuck/confused in those cases.
This is just a rough cut to play with the idea, there's no persistence
of the ProgressEvents, and the init/tick() methods are peppered with
correctness issues. Still, it gives a flavour of how we could add
something friendlier like this to expose simplified progress indicators.
Ideas for further work:
* Add in an MDS handler to capture the progress of an MDS rank as it
goes through replay/reconnect/clientreplay
* A handler for overall cluster restart, that noticed when the mon
quorum was established and all the map timestamps were some time in the
past, and then generated progress based on OSDs coming up and PGs peering.
* Simple: a handler for PG creation after pool creation
* Generate estimated completion times from the rate of progress so far
* Friendlier PGMap output, by hiding all PG states that are explained
by an ongoing ProgressEvent, to only indicate low level PG status for
things that the ProgressEvents don't understand.
Cheers,
John
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
