You could try pausing the upgrade and manually "upgrading" the mds daemons by redeploying them on the new image. Something like "ceph orch daemon redeploy <mds-daemon-name> --image <17.2.6 image>" (daemon names should match those in "ceph orch ps" output). If you do that for all of them and then get them into an up state you should be able to resume the upgrade and have it complete. On Mon, Apr 10, 2023 at 3:25 PM Thomas Widhalm <widhalmt@xxxxxxxxxxxxx> wrote: > Hi, > > If you remember, I hit bug https://tracker.ceph.com/issues/58489 so I > was very relieved when 17.2.6 was released and started to update > immediately. > > But now I'm stuck again with my broken MDS. MDS won't get into up:active > without the update but the update waits for them to get into up:active > state. Seems like a deadlock / chicken-egg problem to me. > > Since I'm still relatively new to Ceph, could you help me? > > What I see when watching the update status: > > { > "target_image": > " > quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635 > ", > "in_progress": true, > "which": "Upgrading all daemon types on all hosts", > "services_complete": [ > "crash", > "mgr", > "mon", > "osd" > ], > "progress": "18/40 daemons upgraded", > "message": "Error: UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect > to host ceph01 at addr (192.168.23.61)", > "is_paused": false > } > > (The offline host was one host that broke during the upgrade. I fixed > that in the meantime and the update went on.) > > And in the log: > > 2023-04-10T19:23:48.750129+0000 mgr.ceph04.qaexpv [INF] Upgrade: Waiting > for mds.mds01.ceph04.hcmvae to be up:active (currently up:replay) > 2023-04-10T19:23:58.758141+0000 mgr.ceph04.qaexpv [WRN] Upgrade: No mds > is up; continuing upgrade procedure to poke things in the right direction > > > Please give me a hint what I can do. > > Cheers, > Thomas > -- > http://www.widhalm.or.at > GnuPG : 6265BAE6 , A84CB603 > Threema: H7AV7D33 > Telegram, Signal: widhalmt@xxxxxxxxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |