We recently upgraded all our clusters to rocky 9.4 and reef 18.2.4. Two of
the clusters show the rgw metrics in the ceph dashboard and the other two
don't. I made sure the firewalls were open for ceph-exporter and that
Prometheus was gathering the stats on all 4 clusters. For the clusters that
are not working, when I do a search on the Prometheus web console at port
9095 on the performance monitoring node using "{instance="rgw1"}, I noticed
that the instance_id shows a long number. I did the same search on the
clusters that show rgw stats in grafana and they have an instance_id that
looks like the drop down in the Grafana dashboard. E.g.
"rgw.rgw.rgw1.biewae".
So it seems that for some reason, on the clusters that are not working, the
ceph-exporter running on the rgw nodes is not pulling the proper instance_id
rgw instance. Is there a way to fix the instance_id that ceph-exporter is
pulling? I can see this instance id when browsing to the servers
http://rgw1:9926/metrics portal. If I run a podman ps, I can see the
container running and it has the unique id .biewae showing.
I used cephadm to enter the ceph-exporter container on the rgw to see what
the logs were saying and I see the same thing repeated:
1 pid path is empty; process metrics won't be fetched for:
ceph-client.rgw.rgw.rgw1.biewae.2.94759457265864
Its only the rgw's that show this error in the ceph-exporter log file, the
mons and osd nodes show no errors in the log and show up in grafana properly
with data. I redeployed ceph-exporter and rgw to the same node in that
order, but no change. I did set the default image container to 18.2.4 prior
to redeploy cause I ran into that issue as well.
Regards,
-Brent
Existing Clusters:
Test: Reef 18.2.0 ( all virtual on nvme )
US Production(HDD): Reef 18.2.4 Cephadm with 11 osd servers, 5 mons, 4 rgw,
2 iscsigw
UK Production(HDD): Reef 18.2.4 Cephadm with 20 osd servers, 5 mons, 4 rgw,
2 iscsigw
US Production(SSD): Reef 18.2.4 Cephadm with 6 osd servers, 5 mons, 4 rgw
UK Production(SSD): Reef 18.2.4 cephadm with 6 osd servers, 5 mons, 4 rgw
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
