Re: calamari dashboard missing usage data after adding/removing ceph nodes

[Michael Kuriger <mk7193@xxxxxx>]

Hi Brian,

I had this problem when I upgraded to firefly (or possibly giant) – At any rate, the data values changed at some point and calamari needs a slight update.

 

Check this file:

/opt/calamari/venv/lib/python2.6/site-packages/calamari_rest_api-0.1-py2.6.egg/calamari_rest/views/v1.py

 

diff v1.py v1.py.ori

105c105

<                 return kb

---

>                 return kb * 1024

111,113c111,113

<             'used_bytes': to_bytes(get_latest_graphite(df_path('total_used_bytes'))),

<             'capacity_bytes': to_bytes(get_latest_graphite(df_path('total_bytes'))),

<             'free_bytes': to_bytes(get_latest_graphite(df_path('total_avail_bytes')))

---

>             'used_bytes': to_bytes(get_latest_graphite(df_path('total_used'))),

>             'capacity_bytes': to_bytes(get_latest_graphite(df_path('total_space'))),

>             'free_bytes': to_bytes(get_latest_graphite(df_path('total_avail')))

 

 

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Brian Jarrett
Sent: Tuesday, December 30, 2014 7:38 AM
To: ceph-users@xxxxxxxx
Subject: [ceph-users] calamari dashboard missing usage data after adding/removing ceph nodes

 

Cephistas,

I've been running a Ceph cluster for several months now.  I started out with a VM called "master" as the admin node and a monitor and two Dell servers as OSD nodes (called  Node1 and Node2) and also made them monitors so I had 3 monitors.

After I got that all running fine, I added Calamari to the admin node.  Then, I needed the Dell servers for another project, so we bought 3 used HP servers and I called them Ceph1, Ceph2, and Ceph3.  The HP servers were added as OSD and Monitor nodes and the Dell servers were removed.  Since I now had 3 monitors with the HP servers, I removed the monitor on "master".

So now "master" is my admin node with Calamari still running on it, but now the "Usage" panel on the dashboard is blank and the two data points for it are not being populated anymore.

So I found:
http://calamari.readthedocs.org/en/latest/operations/troubleshooting.html

I can see data by browsing to my calamari server /graphite/dashboard for all the monitors (there are still metrics for the removed monitors), but I don't see any data under ceph.cluster.<cluster_id>.df.total_used or ...total_avail   Django posts errors in the log about not finding these values, so I think that's where my problem lies.

I've tried to locate all of the graphite and diamond configurations that I can find and I can't locate anything that would populate those values.

Any clues as to where I should look to get this working properly would be great.

Thanks!

Brian

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com