On Wed, Nov 14, 2018 at 3:32 PM Gökhan Kocak <goekhan.kocak@xxxxxxxxxxxxxxxx> wrote: > > Hello everyone, > > we encountered an error with the Prometheus plugin for Ceph mgr: > One osd was down and (therefore) it had no class: > ``` > sudo ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > 28 hdd 7.27539 osd.28 up 1.00000 1.00000 > 6 0 osd.6 down 0 1.00000 > > ``` > > When we tried to curl the metrics, there was an error because the osd > had no class (see below "KeyError: 'class' "). I suspect you're running an old release? This bug (https://tracker.ceph.com/issues/23300) was fixed in 12.2.5. John > Anybody experience the same? > > Isn't this an error on the Prometheus plugin's behalf? When an osd is down, the plugin should not stop working imo. > > ``` > ~> curl -v 127.0.0.1:9283/metrics > * Trying 127.0.0.1... > * Connected to 127.0.0.1 (127.0.0.1) port 9283 (#0) > > GET /metrics HTTP/1.1 > > Host: 127.0.0.1:9283 > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 500 Internal Server Error > < Date: Wed, 14 Nov 2018 13:59:59 GMT > < Content-Length: 1663 > < Content-Type: text/html;charset=utf-8 > < Server: CherryPy/3.5.0 > < > <!DOCTYPE html PUBLIC > "-//W3C//DTD XHTML 1.0 Transitional//EN" > "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd";> > <html> > <head> > <meta http-equiv="Content-Type" content="text/html; > charset=utf-8"></meta> > <title>500 Internal Server Error</title> > <style type="text/css"> > #powered_by { > margin-top: 20px; > border-top: 2px solid black; > font-style: italic; > } > > #traceback { > color: red; > } > </style> > </head> > <body> > <h2>500 Internal Server Error</h2> > <p>The server encountered an unexpected condition which > prevented it from fulfilling the request.</p> > <pre id="traceback">Traceback (most recent call last): > File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line > 670, in respond > response.body = self.handler() > File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line > 217, in __call__ > self.body = self.oldhandler(*args, **kwargs) > File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line > 61, in __call__ > return self.callable(*self.args, **self.kwargs) > File "/usr/lib/x86_64-linux-gnu/ceph/mgr/prometheus/module.py", line > 414, in metrics > metrics = global_instance().collect() > File "/usr/lib/x86_64-linux-gnu/ceph/mgr/prometheus/module.py", line > 351, in collect > self.get_metadata_and_osd_status() > File "/usr/lib/x86_64-linux-gnu/ceph/mgr/prometheus/module.py", line > 310, in get_metadata_and_osd_status > dev_class['class'], > KeyError: 'class' > </pre> > <div id="powered_by"> > <span> > Powered by <a href="http://www.cherrypy.org";>CherryPy 3.5.0</a> > </span> > </div> > </body> > </html> > * Connection #0 to host 127.0.0.1 left intact > ``` > > Kind regards, > > Gökhan > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |