I managed to determine the cause after adding more debugging: https://github.com/containers/podman/issues/8444 https://github.com/pmoogi-redhat/podman/commit/5cbfbd724cc49194d20f7e9b3de3ed65396725de https://github.com/containers/podman/pull/8642 This would appear to impact podman 2.2.0. On Sun, Dec 20, 2020 at 10:23 AM David Orman <ormandj@xxxxxxxxxxxx> wrote: > Hi, > > We have an Octopus installation using cephadm/containers in podman. We're > trying to upgrade from 15.2.7 to 15.2.8. The mgrs upgrade successfully, but > the we then get a failure that cephadm failed to pull an image: > > [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image > > Failed to pull docker.io/ceph/ceph:v15.2.8 on ceph02: > > > Looking at this server, and trying to run the cephadm pull manually: > > root@ceph02:~# cephadm pull > > Using recent ceph image docker.io/ceph/ceph:v15.2.8 > > Pulling container image docker.io/ceph/ceph:v15.2.8... > > Traceback (most recent call last): > > File "/usr/sbin/cephadm", line 6111, in <module> > > r = args.func() > > File "/usr/sbin/cephadm", line 1381, in _infer_image > > return func() > > File "/usr/sbin/cephadm", line 2676, in command_pull > > return command_inspect_image() > > File "/usr/sbin/cephadm", line 1381, in _infer_image > > return func() > > File "/usr/sbin/cephadm", line 2716, in command_inspect_image > > info_from = get_image_info_from_inspect(out.strip(), args.image) > > File "/usr/sbin/cephadm", line 2727, in get_image_info_from_inspect > > image_id, digests = out.split(',', 1) > > ValueError: not enough values to unpack (expected 2, got 1) > > root@ceph02:~# cephadm version > > Using recent ceph image docker.io/ceph/ceph:v15.2.8 > > ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus > (stable) > > root@ceph02:~# podman images > > REPOSITORY TAG IMAGE ID > CREATED SIZE > > docker.io/ceph/ceph v15.2.8 5553b0cb212c 3 > days ago 965 MB > > docker.io/ceph/ceph v15.2.7 2bc420ddb175 2 > weeks ago 979 MB > > docker.io/ceph/ceph v15 2bc420ddb175 2 > weeks ago 979 MB > > docker.io/prom/node-exporter v0.18.1 e5a616e4b9cf 18 > months ago 24.3 MB > > root@ceph02:~# > > > We've upgraded this cluster multiple times prior without issue, so this > appear to be unique to the 15.2.7 -> 15.2.8 upgrade. If we pass --image > with docker.io/ceph/ceph:v15.2.8, the same behavior occurs. It seems like > cephadm in the 15.2.8 image isn't playing nicely with the pull command, for > some reason. > > Interestingly, I did a fresh install of v15.2.8 on one server, and after > the installation, trying a 'cephadm pull' also fails on that 15.2.8 setup, > so it seems like the pull functionality may be broken on these images. > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |