On Mon, Oct 16, 2017 at 2:48 PM, Aaron Bassett <Aaron.Bassett@xxxxxxxxxxxxx> wrote: > Morning, List. > I recently upgraded one of my large (1k osd) clusters from Jewel to Luminous. The upgrade went wonderfully smoothly, and I've now got 3 mgrs running as well. > > One of the things I was most looking forward to getting my hands on was the "native" prometheus exportor. I enabled it as per the documentation, and scraped it: > > time curl sct-r1-r1-head1:9283 | wc -l > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 18.5M 100 18.5M 0 0 424k 0 0:00:44 0:00:44 --:--:-- 297k > 342772 > > real 0m44.715s > user 0m0.224s > sys 0m0.562s Thanks for testing this, that's a useful datapoint. In luminous, the prometheus exporter is exporting all the perf counters -- this is pretty gratuitous. In master, there is new filtering code on the OSD side that limits what gets fed up, based on the priority setting of each perf counter (something like a 10x decrease in the overall volume of counters that get output). There are other things that can be made more efficient here, for example the code is currently fetching the last 10 datapoints for everything and then throwing away everything but the most recent one... John > > > That's ... a *lot* of metrics. Just wondering how large of clusters this has been tested with. It seems to me I'm going to need to turn off some metrics and/or break them into different types of requests to get decent scrape times and small enough scrapes that I can provision a prometheus server to actually be able to ingest it. Has anyone else with a large cluster looked at this yet? > > FWIW, the dashboard plugin performs fairly admirably as long as I don't go to the OSDs page... > > Aaron > CONFIDENTIALITY NOTICE > This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you. > > _______________________________________________ > Ceph-large mailing list > Ceph-large@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-large-ceph.com _______________________________________________ Ceph-large mailing list Ceph-large@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-large-ceph.com
