It looks like the "resource not found" message is being directly output by podman. Is there anything in the cephadm.log (/var/log/ceph/cephadm.log) on one of the hosts where this is happening that says what podman command cephadm was running that hit this error? On Wed, Jan 8, 2025 at 5:27 AM tobias tempel <tobias.tempel@xxxxxxx> wrote: > Dear all, > i'm trying to cephadm-upgrade in an airgapped environment from 18.2.2 to > 18.2.4 ... yet to no avail. > local image registry is a harbor instance, I start the upgrade process with > > ceph orch upgrade start --image harborregistry/quay.io/ceph/ceph:v18.2.4 > > and status looks good > > ceph orch upgrade status > { > "target_image": "harborregistry/quay.io/ceph/ceph:v18.2.4", > "in_progress": true, > "which": "Upgrading all daemon types on all hosts", > "services_complete": [], > "progress": "", > "message": "", > "is_paused": false > } > > In the cephadm log i can see messages like > > cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4', > '--timeout', '895', 'inspect-image'] > > which is fine (works on the commandline), but also > > 2025-01-08 10:33:53,911 7f9c66d50000 INFO /usr/bin/podman: stderr Error: > initializing source docker://harborregistry/quay.io/ceph/ceph:latest: > reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown: > resource not found: repo quay.io/ceph/ceph, tag latest not found > > so for some reason cephadm keeps trying to pull tag "latest" - which i did > not specify - and this fails ... again and again and again. > what am i missing? > can anyone give me a hint, where to look at? > > Thank you very much, > cheers, toBias > > PS: ceph config get mgr > WHO MASK LEVEL OPTION > VALUE RO > mgr basic container_image > harborregistry/quay.io/ceph/ceph * > mgr advanced mgr/cephadm/container_image_alertmanager > harborregistry/quay.io/prometheus/alertmanager * > mgr advanced mgr/cephadm/container_image_base > harborregistry/quay.io/ceph/ceph > mgr advanced mgr/cephadm/container_image_grafana > harborregistry/quay.io/ceph/ceph-grafana * > mgr advanced mgr/cephadm/container_image_node_exporter > harborregistry/quay.io/prometheus/node-exporter * > mgr advanced mgr/cephadm/container_image_prometheus > harborregistry/quay.io/prometheus/prometheus * > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx