Hey there,
i've that problem too, although I got it from updating 17.2.7 to 18.2.4.
After i read this mail I've just fiddled around a bit aaand Prometheus
does not have ceph_osd_recovery_ops.
Then i looked into the files in
/var/lib/ceph/xyz/prometheus.node-name/etc/prometheus/prometheus.yml and
there i found
- job_name: 'ceph-exporter'
honor_labels: true
http_sd_configs:
- url:
http://132.252.90.151:8765/sd/prometheus/sd-config?service=ceph-exporter
There are no ceph-exporter services running on my cluster, only
node-exporters. So i started ceph-exporter services aaand there we can
find ceph_osd_recovery_ops.
I guess they got lost while upgrading? I dont know, but this seems to be
the solution.
ceph orch apply ceph-exporter
ceph orch redeploy prometheus
Best
inDane
On 21.10.24 13:59, Frédéric Nass wrote:
This could be due to a typo in the panel definition in Grafana (comparing the JSON of the working panel with the non-working one might provide more insights) or because the Prometheus Datasource used by Grafana isn't providing any metrics for ceph_osd_recovery_ops.
To check the panel in Grafana, you can go to https://<manager_ip>:3000 then Dashboard / Ceph - Cluster / Recovery Rate / Inspect JSON Panel.
To verify if Prometheus provide some metrics, you can go to https://<manager_ip>:9095 and request metrics for ceph_osd_recovery_ops.
Check [1] if you need to set the Grafana password.
Regards,
Frédéric
[1]https://www.ibm.com/docs/en/storage-ceph/6?topic=access-setting-admin-user-password-grafana
----- Le 21 Oct 24, à 13:09, Sanjay Mohan<sanjaymohan@xxxxxxxxxxxxx> a écrit :
Hi Frédéric,
Thank you for the response.
I tried disabling and re-enabling the module, and it seems that the recovery
metrics are indeed being collected, but still not displayed on the new
dashboard. Interestingly, I have another environment running Ceph 19.2.0 where
the recovery throughput is displayed correctly, but I’m unable to identify any
major differences between the two setups.
Do you have any additional suggestions for troubleshooting this issue further?
Thanks again for your help!
Best regards,
Sanjay Mohan
Software Defined Storage Engineer
sanjaymohan@xxxxxxxxxxxxx
From: Frédéric Nass<frederic.nass@xxxxxxxxxxxxxxxx>
Sent: 21 October 2024 1:22 PM
To: Sanjay Mohan<sanjaymohan@xxxxxxxxxxxxx>
Cc: ceph-users<ceph-users@xxxxxxx>
Subject: Re: Issue with Recovery Throughput Not Visible in Ceph
Dashboard After Upgrade to 19.2.0 (Squid)
Hi Sanjay,
I've just checked the dashboard of a v19.2.0 cluster, and the recovery
throughput is displayed correctly, as shown in the screenshot here [1]. You
might want to consider redeploying the dashboard.
Regards,
Frédéric.
[1] [https://docs.ceph.com/en/latest/mgr/dashboard/ |
https://docs.ceph.com/en/latest/mgr/dashboard/ ]
----- Le 19 Oct 24, à 19:23, Sanjay Mohansanjaymohan@xxxxxxxxxxxxx a écrit :
Dear Ceph Users,
I hope this message finds you well.
I recently performed an upgrade of my Ceph cluster, moving through the following
versions:
* 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid)
After successfully upgrading to Ceph 19.2.0, I noticed an issue where the
recovery throughput is no longer visible in the new Ceph dashboard. However,
Old dashboard metrics and features seem to be working as expected. It is
important to note that the recovery throughput was displayed properly in the
previous version of the Ceph dashboard. I am using Cephadm for the
installation, not Rook.
Current behavior:
* Recovery throughput metrics are not displayed in the new dashboard after
upgrading to 19.2.0.
Expected behavior:
* The recovery throughput should be visible, as it was in previous versions of
the Ceph dashboard.
I am reaching out to inquire if there are any known issues, workarounds, or
upcoming fixes for this. Your assistance in this matter would be greatly
appreciated.
Thank you for your time and support. I look forward to hearing from you soon.
Best regards,
Sanjay Mohan
Software Defined Storage Engineer
sanjaymohan@xxxxxxxxxxxxx
[Amrita University]
Disclaimer: The information transmitted in this email, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. Any views expressed in any message are those of the individual
sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham.
If you received this in error, please contact the sender and destroy any copies
of this information.
_______________________________________________
ceph-users mailing list --ceph-users@xxxxxxx
To unsubscribe send an email toceph-users-leave@xxxxxxx
Disclaimer: The information transmitted in this email, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. Any views expressed in any message are those of the individual
sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham.
If you received this in error, please contact the sender and destroy any copies
of this information.
_______________________________________________
ceph-users mailing list --ceph-users@xxxxxxx
To unsubscribe send an email toceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
