For the last question, cephadm has a config option for whether or not it tries to convert image tags to repo digest (ceph config set mgr mgr/cephadm/use_repo_digest true/false). I'm not sure if setting it to false helps if the tag has already been converted though. In terms of getting the cluster in order, In the case there are actually daemons on this replaced node, if this image doesn't exist anymore you can deploy the individual daemons on the host via "ceph orch daemon redeploy <daemon-name> <image-name>" to whatever 16.2.6 image you want to use for now. They would still be on a slightly different image than the other daemons but if they're the same minor version I imagine it's okay. Once they've been redeployed with a functional image and are up and running and the health warnings go away you can upgrade the cluster to whichever image you were redeploying those daemons with and then they should all end up in line. I do think you would need to add the host back to the cluster first before you could redeploy the daemons in this fashion though. Having the host back in the cluster, even if the daemons are all down, shouldn't cause issues. If the replaced node doesn't actually have any daemons yet, maybe setting the global container image to an image that exists "ceph config set global container_image <image-name>" then adding the host I think should allow you to place daemons on the host as normal. Again, once things are healthy, you can use upgrade to make sure every daemon is on the same image. - Adam King On Fri, Feb 25, 2022 at 10:06 AM Kai Börnert <kai.boernert@xxxxxxxxx> wrote: > Hi, > > what would be the correct way to move forward? > > I have a 3 node cephadm installed cluster, one node died, the other two > are fine and work as expected, so no data loss, but a lot of > remapped/degraded. > > The dead node was replaced and I wanted to add it to the cluster using > "ceph orch host add" > > The current container_image seems to be global: > > quay.io/ceph/ceph@sha256:5d042251e1faa1408663508099cf97b256364300365d403ca5563a518060abac > after some update a while back to 16.2.6. (I did not set the image to a > digest, ceph upgrade did this apparently) > > The new node cannot pull this image, as it no longer exists on quay.io. > > I tried to copy the image via docker save & docker load, however the > digest is not filled due to security reasons this way. > > I kinda do not want to do an additional ceph upgrade until the health is > back at ok. > > > Is there some other way to transfer the image to the new host? > > Is it expected, that images on quay max dissapear at any time? > > Is it possible to force ceph to use a tag instead of a digest? As I > could fix it easily myself then? > > > Greetings, > > Kai > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |