Dear Adam,
thank you very much for your reply.
In /var/log/ceph/cephadm.log i saw lots of entries like this
2025-01-08 10:00:22,045 7ff021d8c000 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'harborregistry/quay.io/ceph/ceph', '--timeout', '895', 'pull']
2025-01-08 10:00:22,172 7ff021d8c000 INFO Pulling container image harborregistry/quay.io/ceph/ceph...
2025-01-08 10:00:27,176 7ff021d8c000 INFO Non-zero exit code 125 from /usr/bin/podman pull harborregistry/quay.io/ceph/ceph
2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr Trying to pull harborregistry/quay.io/ceph/ceph:latest...
2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr time="2025-01-08T10:00:22+01:00" level=warning msg="failed, retrying in 1s ... (1/3). Error: initializing source docker://harborregistry/quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/quay.io/ ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found"
cephadm ['--image', 'harborregistry/quay.io/ceph/ceph', '--timeout', '895', 'pull']
2025-01-08 10:00:22,172 7ff021d8c000 INFO Pulling container image harborregistry/quay.io/ceph/ceph...
2025-01-08 10:00:27,176 7ff021d8c000 INFO Non-zero exit code 125 from /usr/bin/podman pull harborregistry/quay.io/ceph/ceph
2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr Trying to pull harborregistry/quay.io/ceph/ceph:latest...
2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr time="2025-01-08T10:00:22+01:00" level=warning msg="failed, retrying in 1s ... (1/3). Error: initializing source docker://harborregistry/quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/quay.io/ ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found"
...
2025-01-08 10:00:27,176 7ff021d8c000 INFO /usr/bin/podman: stderr Error: initializing source docker://harborregistry/quay.io/ceph/ceph:latest: reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown: resource not found: repo quay.io/ceph/ceph, tag latest not found
2025-01-08 10:00:27,177 7ff021d8c000 ERROR ERROR: Failed command: /usr/bin/podman pull harborregistry/quay.io/ceph/ceph
2025-01-08 10:01:27,459 7f5f185d0000 DEBUG --------------------------------------------------------------------------------
2025-01-08 10:00:27,177 7ff021d8c000 ERROR ERROR: Failed command: /usr/bin/podman pull harborregistry/quay.io/ceph/ceph
2025-01-08 10:01:27,459 7f5f185d0000 DEBUG --------------------------------------------------------------------------------
In the meantime i was given a hint, to
ceph config set mgr container_image harborregistry/quay.io/ceph/ceph:v18.2.4
which indeed changed things to
2025-01-08 17:12:45,952 7ffb1da9b000 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4', '--timeout', '895', 'inspect-image']
2025-01-08 17:12:46,219 7ffb1da9b000 DEBUG /usr/bin/podman: stdout 2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,[quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec
7e1388855ee3d00813b52c3172b0b23b37906 quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77 harborregistry/quay.io/ceph/ceph@sha256:6ac7f923aa1d23
b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 harborregistry/quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77]
2025-01-08 17:12:46,649 7ffb1da9b000 DEBUG ceph: stdout ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
2025-01-08 17:12:50,852 7f21649cf000 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4', '--timeout', '895', 'inspect-image']
2025-01-08 17:12:46,219 7ffb1da9b000 DEBUG /usr/bin/podman: stdout 2bc0b0f4375ddf4270a9a865dfd4e53063acc8e6c3afd7a2546507cafd2ec86a,[quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec
7e1388855ee3d00813b52c3172b0b23b37906 quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77 harborregistry/quay.io/ceph/ceph@sha256:6ac7f923aa1d23
b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 harborregistry/quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77]
2025-01-08 17:12:46,649 7ffb1da9b000 DEBUG ceph: stdout ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)
2025-01-08 17:12:50,852 7f21649cf000 DEBUG --------------------------------------------------------------------------------
Only to now encounter log entries
2025-01-09 00:01:20,077 7fe3a719e000 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'docker.io/ceph/daemon-base:latest-master-devel', '--timeout', '895', '_orch', 'deploy', '--fsid', 'xxxxx']
2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Loaded deploy configuration: {'fsid': 'xxxxx', 'name': 'mon.monitor0x', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_name': 'mon', 'ports': [], 'ip': None, 'deployed_by': ['quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906', 'quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for xxxxx\n[global]\n\tfsid = xxxxx\n\tmon_host = [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0]\n[mon.monitor0x]\npublic network = x.x.x.0/22\n', 'keyring': '[mon.]\n\tkey = xxxxx\n\tcaps mon = "allow *"\n', 'files': {'config': '[mon.monitor0x]\npublic network = x.x.x.0/22\n'}}}
2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Determined image: 'docker.io/ceph/daemon-base:latest-master-devel'
2025-01-09 00:01:20,218 7fe3a719e000 INFO Redeploy daemon mon.monitor0x ...
2025-01-09 00:02:20,255 7fe3a719e000 INFO Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph
2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Trying to pull docker.io/ceph/daemon-base:latest-master-devel...
2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout
2025-01-09 00:02:20,256 7fe3a719e000 ERROR ERROR: Failed to extract uid/gid for path /var/lib/ceph: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph: Trying to pull docker.io/ceph/daemon-base:latest-master-devel...
Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout
cephadm ['--image', 'docker.io/ceph/daemon-base:latest-master-devel', '--timeout', '895', '_orch', 'deploy', '--fsid', 'xxxxx']
2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Loaded deploy configuration: {'fsid': 'xxxxx', 'name': 'mon.monitor0x', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_name': 'mon', 'ports': [], 'ip': None, 'deployed_by': ['quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906', 'quay.io/ceph/ceph@sha256:ac06cdca6f2512a763f1ace8553330e454152b82f95a2b6bf33c3f3ec2eeac77'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for xxxxx\n[global]\n\tfsid = xxxxx\n\tmon_host = [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0] [v2:x.x.x.x:3300/0,v1:x.x.x.x:6789/0]\n[mon.monitor0x]\npublic network = x.x.x.0/22\n', 'keyring': '[mon.]\n\tkey = xxxxx\n\tcaps mon = "allow *"\n', 'files': {'config': '[mon.monitor0x]\npublic network = x.x.x.0/22\n'}}}
2025-01-09 00:01:20,210 7fe3a719e000 DEBUG Determined image: 'docker.io/ceph/daemon-base:latest-master-devel'
2025-01-09 00:01:20,218 7fe3a719e000 INFO Redeploy daemon mon.monitor0x ...
2025-01-09 00:02:20,255 7fe3a719e000 INFO Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph
2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Trying to pull docker.io/ceph/daemon-base:latest-master-devel...
2025-01-09 00:02:20,255 7fe3a719e000 INFO stat: stderr Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout
2025-01-09 00:02:20,256 7fe3a719e000 ERROR ERROR: Failed to extract uid/gid for path /var/lib/ceph: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-master-devel -e NODE_NAME=monitor0x -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/daemon-base:latest-master-devel -c %u %g /var/lib/ceph: Trying to pull docker.io/ceph/daemon-base:latest-master-devel...
Error: initializing source docker://ceph/daemon-base:latest-master-devel: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp 54.236.113.205:443: i/o timeout
After that I was also directed to a config setting "mgr mgr/cephadm/default_registry" which up to now is absent from our configuration (nothing shows up in "ceph config dump | grep registry")
but i yet have no idea what to set here ...
Again thank you very much,
cheers, toBias
cheers, toBias
From: "Adam King" <adking@xxxxxxxxxx>
To: "Tobias Tempel" <tobias.tempel@xxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxx>
Sent: Wednesday, 8 January, 2025 20:15:51
Subject: Re: ceph orch upgrade tries to pull latest?
To: "Tobias Tempel" <tobias.tempel@xxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxx>
Sent: Wednesday, 8 January, 2025 20:15:51
Subject: Re: ceph orch upgrade tries to pull latest?
It looks like the "resource not found" message is being directly output by
podman. Is there anything in the cephadm.log (/var/log/ceph/cephadm.log) on
one of the hosts where this is happening that says what podman command
cephadm was running that hit this error?
On Wed, Jan 8, 2025 at 5:27 AM tobias tempel <tobias.tempel@xxxxxxx> wrote:
> Dear all,
> i'm trying to cephadm-upgrade in an airgapped environment from 18.2.2 to
> 18.2.4 ... yet to no avail.
> local image registry is a harbor instance, I start the upgrade process with
>
> ceph orch upgrade start --image harborregistry/quay.io/ceph/ceph:v18.2.4
>
> and status looks good
>
> ceph orch upgrade status
> {
> "target_image": "harborregistry/quay.io/ceph/ceph:v18.2.4",
> "in_progress": true,
> "which": "Upgrading all daemon types on all hosts",
> "services_complete": [],
> "progress": "",
> "message": "",
> "is_paused": false
> }
>
> In the cephadm log i can see messages like
>
> cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4',
> '--timeout', '895', 'inspect-image']
>
> which is fine (works on the commandline), but also
>
> 2025-01-08 10:33:53,911 7f9c66d50000 INFO /usr/bin/podman: stderr Error:
> initializing source docker://harborregistry/quay.io/ceph/ceph:latest:
> reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown:
> resource not found: repo quay.io/ceph/ceph, tag latest not found
>
> so for some reason cephadm keeps trying to pull tag "latest" - which i did
> not specify - and this fails ... again and again and again.
> what am i missing?
> can anyone give me a hint, where to look at?
>
> Thank you very much,
> cheers, toBias
>
> PS: ceph config get mgr
> WHO MASK LEVEL OPTION
> VALUE RO
> mgr basic container_image
> harborregistry/quay.io/ceph/ceph *
> mgr advanced mgr/cephadm/container_image_alertmanager
> harborregistry/quay.io/prometheus/alertmanager *
> mgr advanced mgr/cephadm/container_image_base
> harborregistry/quay.io/ceph/ceph
> mgr advanced mgr/cephadm/container_image_grafana
> harborregistry/quay.io/ceph/ceph-grafana *
> mgr advanced mgr/cephadm/container_image_node_exporter
> harborregistry/quay.io/prometheus/node-exporter *
> mgr advanced mgr/cephadm/container_image_prometheus
> harborregistry/quay.io/prometheus/prometheus *
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
podman. Is there anything in the cephadm.log (/var/log/ceph/cephadm.log) on
one of the hosts where this is happening that says what podman command
cephadm was running that hit this error?
On Wed, Jan 8, 2025 at 5:27 AM tobias tempel <tobias.tempel@xxxxxxx> wrote:
> Dear all,
> i'm trying to cephadm-upgrade in an airgapped environment from 18.2.2 to
> 18.2.4 ... yet to no avail.
> local image registry is a harbor instance, I start the upgrade process with
>
> ceph orch upgrade start --image harborregistry/quay.io/ceph/ceph:v18.2.4
>
> and status looks good
>
> ceph orch upgrade status
> {
> "target_image": "harborregistry/quay.io/ceph/ceph:v18.2.4",
> "in_progress": true,
> "which": "Upgrading all daemon types on all hosts",
> "services_complete": [],
> "progress": "",
> "message": "",
> "is_paused": false
> }
>
> In the cephadm log i can see messages like
>
> cephadm ['--image', 'harborregistry/quay.io/ceph/ceph:v18.2.4',
> '--timeout', '895', 'inspect-image']
>
> which is fine (works on the commandline), but also
>
> 2025-01-08 10:33:53,911 7f9c66d50000 INFO /usr/bin/podman: stderr Error:
> initializing source docker://harborregistry/quay.io/ceph/ceph:latest:
> reading manifest latest in harborregistry/quay.io/ceph/ceph: unknown:
> resource not found: repo quay.io/ceph/ceph, tag latest not found
>
> so for some reason cephadm keeps trying to pull tag "latest" - which i did
> not specify - and this fails ... again and again and again.
> what am i missing?
> can anyone give me a hint, where to look at?
>
> Thank you very much,
> cheers, toBias
>
> PS: ceph config get mgr
> WHO MASK LEVEL OPTION
> VALUE RO
> mgr basic container_image
> harborregistry/quay.io/ceph/ceph *
> mgr advanced mgr/cephadm/container_image_alertmanager
> harborregistry/quay.io/prometheus/alertmanager *
> mgr advanced mgr/cephadm/container_image_base
> harborregistry/quay.io/ceph/ceph
> mgr advanced mgr/cephadm/container_image_grafana
> harborregistry/quay.io/ceph/ceph-grafana *
> mgr advanced mgr/cephadm/container_image_node_exporter
> harborregistry/quay.io/prometheus/node-exporter *
> mgr advanced mgr/cephadm/container_image_prometheus
> harborregistry/quay.io/prometheus/prometheus *
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
