The only thing I noticed was that I had to change the grafana-api-url
for the dashboard when I stopped one of the two grafana instances. I
wasn't able to test the dashboard before because I had to wait for new
certificates so my browser wouldn't complain about the cephadm cert.
So it seems as if the failover doesn't work entirely automatic, but
it's not too much work to switch the api url. :-)
Zitat von Michael Lipp <mnl@xxxxxx>:
Thank you both very much! I have understood things better now.
I'm not sure, though, whether all URIs are adjusted properly when
changing the placement of the services. Still testing...
Am 08.11.22 um 17:13 schrieb Redouane Kachach Elhichou:
Welcome Eugen,
There are some ongoing efforts to make the whole prometheus stack config
more dynamic by using the http sd configuration [1]. In fact part of the
changes are already in main but they will not be available till the next
Ceph official release.
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config
<https://prometheus.io/docs/prometheus/2.28/configuration/configuration/#http_sd_config>
On Tue, Nov 8, 2022 at 4:47 PM Eugen Block <eblock@xxxxxx> wrote:
I somehow missed the HA part in [1], thanks for pointing that out.
Zitat von Redouane Kachach Elhichou <rkachach@xxxxxxxxxx>:
If you are running quincy and using cephadm then you can have more
instances of prometheus (and other monitoring daemons) running in HA mode
by increasing the number of daemons as in [1]:
from a cephadm shell (to run 2 instances of prometheus and
altertmanager):
ceph orch apply prometheus --placement 'count:2'
ceph orch apply alertmanager --placement 'count:2'
You can have as many instances as you need. You can choose on which nodes
to place them by using the daemon placement specification of cephadm [2]
by
using a specific label for monitoring i.e. In case of mgr failover
cephadm
should reconfigure the daemons accordingly.
[1]
https://docs.ceph.com/en/quincy/cephadm/services/monitoring/#deploying-monitoring-with-cephadm
[2] https://docs.ceph.com/en/quincy/cephadm/services/#daemon-placement
Hope it helps,
Redouane.
On Tue, Nov 8, 2022 at 3:58 PM Eugen Block <eblock@xxxxxx> wrote:
Hi,
the only information I found so far was this statement from the redhat
docs [1]:
When multiple services of the same type are deployed, a
highly-available setup is deployed.
I tried to do that in a virtual test environment (16.2.7) and it seems
to work as expected.
ses7-host1:~ # ceph orch ps --daemon_type prometheus
NAME HOST PORTS STATUS REFRESHED
AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
prometheus.ses7-host1 ses7-host1 running (6h) 12s ago
12M 165M - 2.18.0 8eb9f2694232 04a0b33e2474
prometheus.ses7-host2 ses7-host2 *:9095 host is offline 89s ago
6h 236M - 8eb9f2694232 0cb070cea4eb
host2 was the active mgr before I shut it down, but I still have
access to prometheus metrics as well as active alerts from
alertmanager, there's also one spare instance running, the same
applies for grafana:
ses7-host1:~ # ceph orch ps --daemon_type alertmanager
NAME HOST PORTS STATUS
REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
alertmanager.ses7-host1 ses7-host1 running (6h)
42s ago 12M 33.7M - 0.16.2 903e9b49157e 5a4ffc9a79da
alertmanager.ses7-host2 ses7-host2 *:9093,9094 running (102s)
44s ago 6h 35.5M - 903e9b49157e 71ac3c636a6b
ses7-host1:~ # ceph orch ps --daemon_type prometheus
NAME HOST PORTS STATUS REFRESHED
AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
prometheus.ses7-host1 ses7-host1 running (6h) 44s ago
12M 156M - 2.18.0 8eb9f2694232 04a0b33e2474
prometheus.ses7-host2 ses7-host2 *:9095 running (104s) 47s ago
6h 250M - 8eb9f2694232 87a5a8349f05
ses7-host1:~ # ceph orch ps --daemon_type grafana
NAME HOST PORTS STATUS REFRESHED AGE
MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
grafana.ses7-host1 ses7-host1 running (6h) 47s ago 12M
99.6M - 7.1.5 31b52dc794e2 7935ecf47b38
grafana.ses7-host2 ses7-host2 *:3000 running (107s) 49s ago 6h
108M - 7.1.5 31b52dc794e2 17dea034bb33
I just specified two hosts in the placement section of each service
and deployed them. I think this should be mentioned in the ceph docs
(not only redhat).
[1]
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html/operations_guide/management-of-monitoring-stack-using-the-ceph-orchestrator
Zitat von Michael Lipp <mnl@xxxxxx>:
Hi,
I've just setup a test cluster with cephadm using quincy. Things
work nicely. However, I'm not sure how to "handle" alertmanager and
prometheus.
Both services obviously aren't crucial to the working of the
storage, fine. But there seems to be no built-in fall-over concept.
By default, the active mgr accesses the services using
host.containers.local, thus assuming that they run an the same
machine as the active manager. This assumption is true after the
initial installation. Turning off the host with the active manager
activates the stand-by on another machine, but alertmanager and
prometheus are gone (i.e. not "moved along"). So the active manager
produces lots of error messages when logging into it. Turning the
tuned-off machine on again doesn't help, because alertmanager and
prometheus are back, but on the wrong machine.
I couldn't find anything in the documentation. Are alertmanager and
prometheus supposed to run in some HA-VM? Then I could add the HA-VM
to the cluster with (only) these two services running on it and make
the URIs point to this HA-VM (ceph dashboard
set-alertmanager-api-host ..., ceph dashboard set-grafana-api-url
..., ceph dashboard set-prometheus-api-host...).
How is this supposed to be configured?
- Michael
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
