Dear Eugen, Thanks again for the help. We wanted to go smoothly, as we have unfortunaltey not test clusters, effectively the risk to get a bad version is high, you are right we will see to upgrade to the latest of pacific for the next steps. I have wait about 30 minutes. Still looking why the mgr is crashlooping on the second node Thanks for the help. Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 www.csti.ch -----Original Message----- From: Eugen Block <eblock@xxxxxx> Sent: mercredi, 6 mars 2024 10:33 To: ceph-users@xxxxxxx Subject: Re: Upgarde from 16.2.1 to 16.2.2 pacific stuck Hi, a couple of things. First, is there any specific reason why you're upgrading from .1 to .2? Why not directly to .15? It seems unnecessary and you're risking upgrading to a "bad" version (I believe it was 16.2.7) if you're applying evey minor release. Or why not upgrading to Quincy or Reef directly? Second, the error message has changed (https://github.com/ceph/ceph/pull/41257) from "could not verify host allowed virtual ips" to "does not belong to mon public_network". I saw this just recently during an upgrade of a cluster I didn't deploy and it turned out to be a misconfiguration issue (mismatch between cephadm ssh-user and missing pub key). I recommend to verify the ssh connections. ceph cephadm get-ssh-config ceph cephadm get-user ceph cephadm get-pub-key It also could be just a timing issue according to Sage's statement in above PR: > Oh! I know what the problem is. 1897d1c changed the way we store the > per-host network interface/network info. > On upgrade, cephadm thinks > there are no networks on each host until the device refresh happens. How long did you wait? Zitat von Edouard FAZENDA <e.fazenda@xxxxxxx>: > Dear Ceph Community, > > > > I am in the process of upgrading ceph pacific 16.2.1 to 16.2.2 , I > have followed the documentation : > https://docs.ceph.com/en/pacific/cephadm/upgrade/ > > > > My cluster is in Healthy state , but the upgrade is not going forward > , as on the cephadm logs I have the following : > > > > # Ceph -W cephadm > > 2024-03-06T08:39:11.653447+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: > Need to upgrade myself (mgr.rke-sh1-1.qskoyj) > > 2024-03-06T08:39:12.281386+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: > Updating mgr.rke-sh1-2.lxmguj > > 2024-03-06T08:39:12.286096+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying > daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 > > 2024-03-06T08:39:19.347877+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered > out host > rke-sh1-1: could not verify host allowed virtual ips > > 2024-03-06T08:39:19.347989+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered > out host > rke-sh1-3: could not verify host allowed virtual ips > > 2024-03-06T08:39:19.366355+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: > Need to upgrade myself (mgr.rke-sh1-1.qskoyj) > > 2024-03-06T08:39:19.965822+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: > Updating mgr.rke-sh1-2.lxmguj > > 2024-03-06T08:39:19.969089+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying > daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 > > 2024-03-06T08:39:26.961455+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered > out host > rke-sh1-1: could not verify host allowed virtual ips > > 2024-03-06T08:39:26.961502+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered > out host > rke-sh1-3: could not verify host allowed virtual ips > > 2024-03-06T08:39:26.973897+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: > Need to upgrade myself (mgr.rke-sh1-1.qskoyj) > > 2024-03-06T08:39:27.623773+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: > Updating mgr.rke-sh1-2.lxmguj > > 2024-03-06T08:39:27.628115+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying > daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 > > > > My public_network is set : > > > > root@rke-sh1-1:~# ceph config dump | grep public_network > > mon advanced public_network > 10.10.71.0/24 > > * > > Do you have an idea why I have the following error : > > > > Filtered out host: could not verify host allowed virtual ips > > > > > > Current state of the upgrade : > > > > # ceph orch upgrade status > > { > > "target_image": > "docker.io/ceph/ceph@sha256:8cdd8c7dfc7be5865255f0d59c048a1fb8d1335f69 > 23996e > 2c2d9439499b5cf2", > > "in_progress": true, > > "services_complete": [], > > "progress": "0/35 ceph daemons upgraded", > > "message": "Currently upgrading mgr daemons" > > } > > > > progress: > > Upgrade to 16.2.2 (24m) > > [............................] > > > > Thanks for the help. > > > > Best Regards, > > > > Edouard FAZENDA > > Technical Support > > > > > > > > Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 > > > > <https://www.csti.ch/> www.csti.ch _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
