Re: Luminous Prometheus exporter on Large clusters

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On Mon, Oct 16, 2017 at 2:48 PM, Aaron Bassett
<Aaron.Bassett@xxxxxxxxxxxxx> wrote:
> Morning, List.
> I recently upgraded one of my large (1k osd) clusters from Jewel to Luminous. The upgrade went wonderfully smoothly, and I've now got 3 mgrs running as well.
> One of the things I was most looking forward to getting my hands on was the "native" prometheus exportor. I enabled it as per the documentation, and scraped it:
> time curl sct-r1-r1-head1:9283 | wc -l
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100 18.5M  100 18.5M    0     0   424k      0  0:00:44  0:00:44 --:--:--  297k
>   342772
> real    0m44.715s
> user    0m0.224s
> sys     0m0.562s

Thanks for testing this, that's a useful datapoint.

In luminous, the prometheus exporter is exporting all the perf
counters -- this is pretty gratuitous.  In master, there is new
filtering code on the OSD side that limits what gets fed up, based on
the priority setting of each perf counter (something like a 10x
decrease in the overall volume of counters that get output).

There are other things that can be made more efficient here, for
example the code is currently fetching the last 10 datapoints for
everything and then throwing away everything but the most recent


> That's ... a *lot* of metrics. Just wondering how large of clusters this has been tested with. It seems to me I'm going to need to turn off some metrics and/or break them into different types of requests to get decent scrape times and small enough scrapes that I can provision a prometheus server to actually be able to ingest it. Has anyone else with a large cluster looked at this yet?
> FWIW, the dashboard plugin performs fairly admirably as long as I don't go to the OSDs page...
> Aaron
> This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
> _______________________________________________
> Ceph-large mailing list
> Ceph-large@xxxxxxxxxxxxxx
Ceph-large mailing list

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFS]

  Powered by Linux