Re: Issue with Recovery Throughput Not Visible in Ceph Dashboard After Upgrade to 19.2.0 (Squid)

[Eugen Block <eblock@xxxxxx>]

You're right, deploying ceph-exporter fixes the recovery graph in my  
lab cluster. I didn't have to redeploy prometheus.

Thanks!

Zitat von mailing-lists <mailing-lists@xxxxxxxxx>:

> Hey there,
>
> i've that problem too, although I got it from updating 17.2.7 to  
> 18.2.4. After i read this mail I've just fiddled around a bit aaand  
> Prometheus does not have ceph_osd_recovery_ops.
>
> Then i looked into the files in  
> /var/lib/ceph/xyz/prometheus.node-name/etc/prometheus/prometheus.yml  
> and there i found
>
>   - job_name: 'ceph-exporter'
>     honor_labels: true
>     http_sd_configs:
>     - url:  
> http://132.252.90.151:8765/sd/prometheus/sd-config?service=ceph-exporter
>
> There are no ceph-exporter services running on my cluster, only  
> node-exporters. So i started ceph-exporter services aaand there we  
> can find ceph_osd_recovery_ops.
>
> I guess they got lost while upgrading? I dont know, but this seems  
> to be the solution.
>
>
> ceph orch apply ceph-exporter
>
> ceph orch redeploy prometheus
>
>
> Best
>
> inDane
>
>
>
>
>
>
>
> On 21.10.24 13:59, Frédéric Nass wrote:
> > This could be due to a typo in the panel definition in Grafana  
> > (comparing the JSON of the working panel with the non-working one  
> > might provide more insights) or because the Prometheus Datasource  
> > used by Grafana isn't providing any metrics for  
> > ceph_osd_recovery_ops.
> >
> > To check the panel in Grafana, you can go to  
> > https://<manager_ip>:3000 then Dashboard / Ceph - Cluster /  
> > Recovery Rate / Inspect JSON Panel.
> > To verify if Prometheus provide some metrics, you can go to  
> > https://<manager_ip>:9095 and request metrics for  
> > ceph_osd_recovery_ops.
> >
> > Check [1] if you need to set the Grafana password.
> >
> > Regards,
> > Frédéric
> >
> > [1]https://www.ibm.com/docs/en/storage-ceph/6?topic=access-setting-admin-user-password-grafana ----- Le 21 Oct 24, à 13:09, Sanjay Mohan<sanjaymohan@xxxxxxxxxxxxx> a écrit  
> > :
> >
> > > Hi Frédéric,
> > > Thank you for the response.
> > > I tried disabling and re-enabling the module, and it seems that  
> > > the recovery
> > > metrics are indeed being collected, but still not displayed on the new
> > > dashboard. Interestingly, I have another environment running Ceph  
> > > 19.2.0 where
> > > the recovery throughput is displayed correctly, but I’m unable to  
> > > identify any
> > > major differences between the two setups.
> > > Do you have any additional suggestions for troubleshooting this  
> > > issue further?
> > > Thanks again for your help!
> > > Best regards,
> > > Sanjay Mohan
> > > Software Defined Storage Engineer
> > > sanjaymohan@xxxxxxxxxxxxx
> > > From: Frédéric Nass<frederic.nass@xxxxxxxxxxxxxxxx>
> > > Sent: 21 October 2024 1:22 PM
> > > To: Sanjay Mohan<sanjaymohan@xxxxxxxxxxxxx>
> > > Cc: ceph-users<ceph-users@xxxxxxx>
> > > Subject: Re:  Issue with Recovery Throughput Not  
> > > Visible in Ceph
> > > Dashboard After Upgrade to 19.2.0 (Squid)
> > > Hi Sanjay,
> > > I've just checked the dashboard of a v19.2.0 cluster, and the recovery
> > > throughput is displayed correctly, as shown in the screenshot here [1]. You
> > > might want to consider redeploying the dashboard.
> > > Regards,
> > > Frédéric.
> > > [1] [https://docs.ceph.com/en/latest/mgr/dashboard/ |
> > > https://docs.ceph.com/en/latest/mgr/dashboard/ ]
> > > ----- Le 19 Oct 24, à 19:23, Sanjay Mohansanjaymohan@xxxxxxxxxxxxx  
> > > a écrit :
> > > > Dear Ceph Users,
> > > > I hope this message finds you well.
> > > > I recently performed an upgrade of my Ceph cluster, moving  
> > > > through the following
> > > > versions:
> > > > * 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid)
> > > > After successfully upgrading to Ceph 19.2.0, I noticed an issue where the
> > > > recovery throughput is no longer visible in the new Ceph  
> > > > dashboard. However,
> > > > Old dashboard metrics and features seem to be working as expected. It is
> > > > important to note that the recovery throughput was displayed  
> > > > properly in the
> > > > previous version of the Ceph dashboard. I am using Cephadm for the
> > > > installation, not Rook.
> > > > Current behavior:
> > > > * Recovery throughput metrics are not displayed in the new dashboard after
> > > > upgrading to 19.2.0.
> > > > Expected behavior:
> > > > * The recovery throughput should be visible, as it was in  
> > > > previous versions of
> > > > the Ceph dashboard.
> > > > I am reaching out to inquire if there are any known issues,  
> > > > workarounds, or
> > > > upcoming fixes for this. Your assistance in this matter would be greatly
> > > > appreciated.
> > > > Thank you for your time and support. I look forward to hearing  
> > > > from you soon.
> > > > Best regards,
> > > > Sanjay Mohan
> > > > Software Defined Storage Engineer
> > > > sanjaymohan@xxxxxxxxxxxxx
> > > > [Amrita University]
> > > > Disclaimer: The information transmitted in this email, including  
> > > > attachments, is
> > > > intended only for the person(s) or entity to which it is addressed and may
> > > > contain confidential and/or privileged material. Any review,  
> > > > retransmission,
> > > > dissemination or other use of, or taking of any action in  
> > > > reliance upon this
> > > > information by persons or entities other than the intended recipient is
> > > > prohibited. Any views expressed in any message are those of the individual
> > > > sender and may not necessarily reflect the views of Amrita Vishwa  
> > > > Vidyapeetham.
> > > > If you received this in error, please contact the sender and  
> > > > destroy any copies
> > > > of this information.
> > > > _______________________________________________
> > > > ceph-users mailing list --ceph-users@xxxxxxx
> > > > To unsubscribe send an email toceph-users-leave@xxxxxxx
> > > Disclaimer: The information transmitted in this email, including  
> > > attachments, is
> > > intended only for the person(s) or entity to which it is addressed and may
> > > contain confidential and/or privileged material. Any review,  
> > > retransmission,
> > > dissemination or other use of, or taking of any action in reliance  
> > > upon this
> > > information by persons or entities other than the intended recipient is
> > > prohibited. Any views expressed in any message are those of the individual
> > > sender and may not necessarily reflect the views of Amrita Vishwa  
> > > Vidyapeetham.
> > > If you received this in error, please contact the sender and  
> > > destroy any copies
> > > of this information.
> > _______________________________________________
> > ceph-users mailing list --ceph-users@xxxxxxx
> > To unsubscribe send an email toceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx