I think with the `config set` commands there is logic to notify the relevant mgr modules and update their values. That might not exist with `config rm`, so it's still using the last set value. Looks like a real bug. Curious what happens if the mgr restarts after the `config rm`. Whether it goes back to the default image in that case or not. Might take a look later. On Mon, May 15, 2023 at 7:37 AM Daniel Krambrock < krambrock@xxxxxxxxxxxxxxxxxx> wrote: > Hello. > > I think i found a bug in cephadm/ceph orch: > Redeploying a container image (tested with alertmanager) after removing > a custom `mgr/cephadm/container_image_alertmanager` value, deploys the > previous container image and not the default container image. > > I'm running `cephadm` from ubuntu 22.04 pkg 17.2.5-0ubuntu0.22.04.3 and > `ceph` version 17.2.6. > > Here is an example. Node clrz20-08 is the node altermanager is running > on, clrz20-01 the node I'm controlling ceph from: > > * Get alertmanager version > ``` > root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == > "alertmanager")| .container_image_name' > "quay.io/prometheus/alertmanager:v0.23.0" > ``` > > * Set alertmanager image > ``` > root@clrz20-01:~# ceph config set mgr > mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager > root@clrz20-01:~# ceph config get mgr > mgr/cephadm/container_image_alertmanager > quay.io/prometheus/alertmanager > ``` > > * redeploy altermanager > ``` > root@clrz20-01:~# ceph orch redeploy alertmanager > Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' > ``` > > * Get alertmanager version > ``` > root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == > "alertmanager")| .container_image_name' > "quay.io/prometheus/alertmanager:latest" > ``` > > * Remove alertmanager image setting, revert to default: > ``` > root@clrz20-01:~# ceph config rm mgr > mgr/cephadm/container_image_alertmanager > root@clrz20-01:~# ceph config get mgr > mgr/cephadm/container_image_alertmanager > quay.io/prometheus/alertmanager:v0.23.0 > ``` > > * redeploy altermanager > ``` > root@clrz20-01:~# ceph orch redeploy alertmanager > Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' > ``` > > * Get alertmanager version > ``` > root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == > "alertmanager")| .container_image_name' > "quay.io/prometheus/alertmanager:latest" > ``` > -> `mgr/cephadm/container_image_alertmanager` is set to > `quay.io/prometheus/alertmanager:v0.23.0` > <http://quay.io/prometheus/alertmanager:v0.23.0>, but redeploy uses > `quay.io/prometheus/alertmanager:latest` > <http://quay.io/prometheus/alertmanager:latest>. This looks like a bug. > > * Set alertmanager image explicitly to the default value > ``` > root@clrz20-01:~# ceph config set mgr > mgr/cephadm/container_image_alertmanager > quay.io/prometheus/alertmanager:v0.23.0 > root@clrz20-01:~# ceph config get mgr > mgr/cephadm/container_image_alertmanager > quay.io/prometheus/alertmanager:v0.23.0 > ``` > > * redeploy altermanager > ``` > root@clrz20-01:~# ceph orch redeploy alertmanager > Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' > ``` > > * Get alertmanager version > ``` > root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == > "alertmanager")| .container_image_name' > "quay.io/prometheus/alertmanager:v0.23.0" > ``` > -> Setting `mgr/cephadm/container_image_alertmanager` to the default > setting fixes the issue. > > > > Bests, > Daniel > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |