Seems like a BUG in cephadm, the ceph-exporter when deployed doesn't
specify its port that's why it's not being opened automatically. You can
see that in the cephadm logs (ports list is empty):
2024-09-09 04:39:48,986 7fc2993d7740 DEBUG Loaded deploy configuration:
{'fsid': '250b9d7c-6e65-11ef-8e0e-525400ecf80a', 'name':
'ceph-exporter.ceph-node-0', 'image': '', 'deploy_arguments': [], 'params':
{}, 'meta': {'service_name': 'ceph-exporter', *'ports': [],* 'ip': None,
'deployed_by': ['
quay.ceph.io/ceph-ci/ceph@sha256:02ce7c1aa356b524041713a3603da8445c4fe00ed30cb1c1f91532926db20d3c'],
'rank': None, 'rank_generation': None,
I opened the following tracker to fix the issue:
https://tracker.ceph.com/issues/67975
On Mon, Sep 9, 2024 at 2:54 PM Sake Ceph <ceph@xxxxxxxxxxx> wrote:
> We're using RHEL 8 and 9 and on both the port was not open.
> It's just strange it isn't working for ceph-exporter but just fine for
> everything else.
>
> Kind regards,
> Sake
>
> > Op 09-09-2024 14:03 CEST schreef Eugen Block <eblock@xxxxxx>:
> >
> >
> > Those two daemons are handled differently by cephadm, they're
> > different classes (grafana is "class Monitoring(ContainerDaemonForm)"
> > while ceph-exporter is "class CephExporter(ContainerDaemonForm)"),
> > therefore they have different metadata etc., for example:
> >
> > soc9-ceph:~ # jq '.ports'
> > /var/lib/ceph/{FSID}/ceph-exporter.soc9-ceph/unit.meta
> > []
> >
> > soc9-ceph:~ # jq '.ports'
> /var/lib/ceph/{FSID}/grafana.soc9-ceph/unit.meta
> > [
> > 3000
> > ]
> >
> > But that's about all I can provide here. Maybe the host OS plays some
> > role here as well, not sure.
> >
> > Zitat von Sake Ceph <ceph@xxxxxxxxxxx>:
> >
> > > We're using default :) I'm talking about the deployment
> > > configuration which is shown in the log files when deploying
> > > grafana/ceph-exporter.
> > >
> > > I got the same configuration as you for ceph-exporter (the default)
> > > when exporting the service.
> > >
> > > Kind regards,
> > > Sake
> > >
> > >> Op 09-09-2024 12:04 CEST schreef Eugen Block <eblock@xxxxxx>:
> > >>
> > >>
> > >> Can you be more specific about "deploy configuration"? Do you have
> > >> your own spec files for grafana and ceph-exporter?
> > >> I just ran 'ceph orch apply ceph-exporter' and the resulting config is
> > >> this one:
> > >>
> > >> # ceph orch ls ceph-exporter --export
> > >> service_type: ceph-exporter
> > >> service_name: ceph-exporter
> > >> placement:
> > >> host_pattern: '*'
> > >> spec:
> > >> prio_limit: 5
> > >> stats_period: 5
> > >>
> > >> Zitat von Sake Ceph <ceph@xxxxxxxxxxx>:
> > >>
> > >> > Hello Eugen,
> > >> >
> > >> > Well nothing about enabling port 9926.
> > >> >
> > >> > For example I see the following when deploying Grafanan:
> > >> > 2024-09-05 14:27:30,969 7fd2e6583740 INFO firewalld ready
> > >> > 2024-09-05 14:27:31,334 7fd2e6583740 DEBUG /bin/firewall-cmd:
> > >> stdout success
> > >> > 2024-09-05 14:27:31,350 7fd2e6583740 INFO firewalld ready
> > >> > 2024-09-05 14:27:31,593 7fd2e6583740 DEBUG Non-zero exit code 1 from
> > >> > /bin/firewall-cmd --permanent --query-port 3000/tcp
> > >> > 2024-09-05 14:27:31,594 7fd2e6583740 DEBUG /bin/firewall-cmd:
> stdout no
> > >> > 2024-09-05 14:27:31,594 7fd2e6583740 INFO Enabling firewalld port
> > >> > 3000/tcp in current zone...
> > >> > 2024-09-05 14:27:31,832 7fd2e6583740 DEBUG /bin/firewall-cmd:
> > >> stdout success
> > >> > 2024-09-05 14:27:32,212 7fd2e6583740 DEBUG /bin/firewall-cmd:
> > >> stdout success
> > >> >
> > >> > But only the following when deploying ceph-exporter:
> > >> > 2024-09-05 12:17:48,897 7f3d7cc0e740 INFO firewalld ready
> > >> > 2024-09-05 12:17:49,269 7f3d7cc0e740 DEBUG /bin/firewall-cmd:
> > >> stdout success
> > >> >
> > >> > When looking in the deploy configuration, Grafana shows 'ports':
> > >> > [3000], but ceph-exporter shows 'ports': []
> > >> >
> > >> > Kind regards,
> > >> > Sake
> > >> >
> > >> >> Op 09-09-2024 10:50 CEST schreef Eugen Block <eblock@xxxxxx>:
> > >> >>
> > >> >>
> > >> >> Sorry, clicked "send" too soon. In a test cluster, cephadm.log
> shows
> > >> >> that it would try to open a port if a firewall was enabled:
> > >> >>
> > >> >> 2024-09-09 10:48:59,686 7f3142b11740 DEBUG firewalld.service is
> > >> not enabled
> > >> >> 2024-09-09 10:48:59,686 7f3142b11740 DEBUG Not possible to enable
> > >> >> service <ceph-exporter>. firewalld.service is not available
> > >> >>
> > >> >> Zitat von Eugen Block <eblock@xxxxxx>:
> > >> >>
> > >> >> > Do you see anything in the cephadm.log related to the firewall?
> > >> >> >
> > >> >> > Zitat von Sake Ceph <ceph@xxxxxxxxxxx>:
> > >> >> >
> > >> >> >> After opening port 9926 manually, the Grafana dashboards show
> > >> the data.
> > >> >> >> So is this a bug?
> > >> >> >>
> > >> >> >> Kind regards,
> > >> >> >> Sake
> > >> >> >>> Op 06-09-2024 17:39 CEST schreef Sake Ceph <ceph@xxxxxxxxxxx>:
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> That is working, but I noticed the firewall isn't opened for
> that
> > >> >> >>> port. Shouldn't cephadm manage this, like it does for all the
> > >> >> >>> other ports?
> > >> >> >>>
> > >> >> >>> Kind regards,
> > >> >> >>> Sake
> > >> >> >>>
> > >> >> >>>> Op 06-09-2024 16:14 CEST schreef Björn Lässig
> > >> >> <b.laessig@xxxxxxxxxxxxxx>:
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> Am Mittwoch, dem 04.09.2024 um 20:01 +0200 schrieb Sake Ceph:
> > >> >> >>>> > After the upgrade from 17.2.7 to 18.2.4 a lot of graphs are
> > >> >> empty. For
> > >> >> >>>> > example the Osd latency under OSD device details or the
> > >> Osd Overview
> > >> >> >>>> > has a lot of No data messages.
> > >> >> >>>> >
> > >> >> >>>>
> > >> >> >>>> is the ceph-exporter listening on port 9926 (on every host)?
> > >> >> >>>>
> > >> >> >>>> ss -tlpn sport 9926
> > >> >> >>>>
> > >> >> >>>> Can you connect via browser?
> > >> >> >>>>
> > >> >> >>>> curl localhost:9926/metrics
> > >> >> >>>>
> > >> >> >>>> > I deployed ceph-exporter on all hosts, am I missing
> something? Did
> > >> >> >>>> > even a redeploy of prometheus.
> > >> >> >>>>
> > >> >> >>>> there is a bug, that this exporter does not listens for IPv6.
> > >> >> >>>>
> > >> >> >>>> greetings
> > >> >> >>>> Björn
> > >> >> >>>> _______________________________________________
> > >> >> >>>> ceph-users mailing list -- ceph-users@xxxxxxx
> > >> >> >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >> >> >>> _______________________________________________
> > >> >> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> > >> >> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >> >> >> _______________________________________________
> > >> >> >> ceph-users mailing list -- ceph-users@xxxxxxx
> > >> >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >> >>
> > >> >>
> > >> >> _______________________________________________
> > >> >> ceph-users mailing list -- ceph-users@xxxxxxx
> > >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
