Re: Upgrade from 17.2.5 to 17.2.6 stuck at MDS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 4/11/23 15:59, Thomas Widhalm wrote:


On 11.04.23 09:16, Xiubo Li wrote:

On 4/11/23 03:24, Thomas Widhalm wrote:
Hi,

If you remember, I hit bug https://tracker.ceph.com/issues/58489 so I was very relieved when 17.2.6 was released and started to update immediately.

Please note, this fix is not in the v17.2.6 yet in upstream code.


Thanks for the information. I misread the information in the tracker. Do you have a predicted schedule for the backport? Or should I go for a specific pre-release? I don't want to take chances but I'm desperate because my production system is affected and offline for several weeks now.

The backport is already queued to review and test, but I am not very sure when it can get merged. I am not very sure this can 100% resolve your issue in case when you have other corruptions in you production.

Thanks

Thanks,
Thomas

Thanks

- Xiubo


But now I'm stuck again with my broken MDS. MDS won't get into up:active without the update but the update waits for them to get into up:active state. Seems like a deadlock / chicken-egg problem to me.

Since I'm still relatively new to Ceph, could you help me?

What I see when watching the update status:

{
    "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635",
    "in_progress": true,
    "which": "Upgrading all daemon types on all hosts",
    "services_complete": [
        "crash",
        "mgr",
    "mon",
    "osd"
    ],
    "progress": "18/40 daemons upgraded",
    "message": "Error: UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host ceph01 at addr (192.168.23.61)",
    "is_paused": false
}

(The offline host was one host that broke during the upgrade. I fixed that in the meantime and the update went on.)

And in the log:

2023-04-10T19:23:48.750129+0000 mgr.ceph04.qaexpv [INF] Upgrade: Waiting for mds.mds01.ceph04.hcmvae to be up:active (currently up:replay) 2023-04-10T19:23:58.758141+0000 mgr.ceph04.qaexpv [WRN] Upgrade: No mds is up; continuing upgrade procedure to poke things in the right direction


Please give me a hint what I can do.

Cheers,
Thomas

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux