On 15/06/2015 14:52, Sage Weil wrote:
I seem to remember having a short conversation about something like this a
few CDS's back... although I think it was 'rados top'. IIRC the basic
idea we had was for each OSD to track it's top clients (using some
approximate LRU type algorithm) and then either feed this relatively small
amount of info (say, top 10-100 clients) back to the mon for summation,
or dump via the admin socket for calamari to aggregate.
This doesn't give you the rbd image name, but I bet we could infer that
without too much trouble (e.g., include a recent object or two with the
client). Or, just assume that client id is enough (it'll include an IP
and PID... enough info to find the /var/run/ceph admin socket or the VM
process.
If we were going to do top clients, I think it'd make sense to also have a
top objects list as well, so you can see what the hottest objects in the
cluster are.
The following is a bit of a tangent...
A few weeks ago I was thinking about general solutions to this problem
(for the filesystem). I played with (very briefly on wip-live-query)
the idea of publishing a list of queries to the MDSs/OSDs, that would
allow runtime configuration of what kind of thing we're interested in
and how we want it broken down.
If we think of it as an SQL-like syntax, then for the RBD case we would
have something like:
SELECT read_bytes, write_bytes WHERE pool=rbd GROUP BY rbd_image
(You'd need a protocol-specific module of some kind to define what
"rbd_image" meant here, which would do a simple mapping from object
attributes to an identifier (similar would exist for e.g. cephfs inode))
Each time an OSD does an operation, it consults the list of active
"performance queries" and updates counters according to the value of the
GROUP BY parameter for the query (so the above example each OSD would be
keeping a result row for each rbd image touchd).
The LRU part could be implemented as LIMIT BY + SORT parameters, such
that the result rows would be periodically sorted and the least-touched
results would drop off the list. That would probably be used in
conjunction with a decay operator on the sorted-by field, like:
SELECT read_bytes, write_bytes,ops WHERE pool=rbd GROUP BY rbd_image
SORT BY movingAverage(derivative(ops)) LIMIT 100
Combining WHERE clauses would let the user "drill down" (apologies for
buzzword) by doing things like identifying the most busy clients, and
then for each of those clients identify which images/files/objects the
client is most active on, or vice versa identify busy objects and then
see which clients are hitting them. Usually keeping around enough stats
to enable this is prohibitive at scale, but it's fine when you're
actively creating custom queries for the results you're really
interested in, instead of keeping N_clients*N_objects stats, and when
you have the LIMIT part to ensure results never get oversized.
The GROUP BY options would also include metadata sent from clients, e.g.
the obvious cases like VM instance names, or rack IDs, or HPC job IDs.
Maybe also some less obvious ones like decorating cephfs IOs with the
inode of the directory containing the file, so that OSDs could
accumulate per-directory bandwidth numbers, and user could ask "which
directory is bandwidth-hottest?" as well as "which file is
bandwidth-hottest?".
Then, after implementing all that craziness, you get some kind of wild
multicolored GUI that shows you where the action is in your system at a
cephfs/rgw/rbd level.
Cheers,
John
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
