The process has now started but I have the following error on mgr to the second node root@rke-sh1-1:~# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID crash.rke-sh1-1 rke-sh1-1 running (12d) 41s ago 12d 16.2.1 c757e4a3636b e8652edb2b49 crash.rke-sh1-2 rke-sh1-2 running (12d) 2s ago 20M 16.2.1 c757e4a3636b a1249a605ee0 crash.rke-sh1-3 rke-sh1-3 running (12d) 41s ago 12d 16.2.1 c757e4a3636b 026667bc1776 mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (12d) 41s ago 5M 16.2.1 c757e4a3636b 9b4c2b08b759 mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (12d) 2s ago 23M 16.2.1 c757e4a3636b 71681a5f34d3 mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (12d) 41s ago 4M 16.2.1 c757e4a3636b e89946ad6b7e mgr.rke-sh1-1.qskoyj rke-sh1-1 *:8082,9283 running (66m) 41s ago 2y 16.2.2 5e237c38caa6 123cabbc2994 mgr.rke-sh1-2.lxmguj rke-sh1-2 *:8082,9283 running (6s) 2s ago 22M 16.2.2 5e237c38caa6 b2a9047be1d6 mgr.rke-sh1-3.ckunvo rke-sh1-3 *:8082,9283 running (12d) 41s ago 7M 16.2.1 c757e4a3636b 2fcaf18f3218 mon.rke-sh1-1 rke-sh1-1 running (37m) 41s ago 37m 16.2.1 c757e4a3636b 84e63e0415a8 mon.rke-sh1-2 rke-sh1-2 running (12d) 2s ago 4M 16.2.1 c757e4a3636b f4b32ba4466b mon.rke-sh1-3 rke-sh1-3 running (12d) 41s ago 12d 16.2.1 c757e4a3636b d5e44c245998 osd.0 rke-sh1-2 running (12d) 2s ago 3y 16.2.1 c757e4a3636b 7b0e69942c15 osd.1 rke-sh1-3 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 4451654d9a2d osd.10 rke-sh1-3 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 3f9d5f95e284 osd.11 rke-sh1-1 running (12d) 41s ago 3y 16.2.1 c757e4a3636b db1cc6d2e37f osd.12 rke-sh1-2 running (12d) 2s ago 3y 16.2.1 c757e4a3636b de416c1ef766 osd.13 rke-sh1-3 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 25a281cc5a9b osd.14 rke-sh1-1 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 62f25ba61667 osd.15 rke-sh1-2 running (12d) 2s ago 3y 16.2.1 c757e4a3636b d3514d823c45 osd.16 rke-sh1-3 running (12d) 41s ago 3y 16.2.1 c757e4a3636b bba857759bfe osd.17 rke-sh1-1 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 59281d4bb3d0 osd.2 rke-sh1-1 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 418041b5e60d osd.3 rke-sh1-2 running (12d) 2s ago 3y 16.2.1 c757e4a3636b 04a0e29d5623 osd.4 rke-sh1-1 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 1cc78a5153d3 osd.5 rke-sh1-3 running (12d) 41s ago 3y 16.2.1 c757e4a3636b 39a4b11e31fb osd.6 rke-sh1-2 running (12d) 2s ago 3y 16.2.1 c757e4a3636b 2f218ffb566e osd.7 rke-sh1-1 running (12d) 41s ago 3y 16.2.1 c757e4a3636b cf761fbe4d5f osd.8 rke-sh1-3 running (12d) 41s ago 3y 16.2.1 c757e4a3636b f9f85480e800 osd.9 rke-sh1-2 running (12d) 2s ago 3y 16.2.1 c757e4a3636b 664c54ff46d2 rgw.default.rke-sh1-1.dgucwl rke-sh1-1 *:8000 running (12d) 41s ago 22M 16.2.1 c757e4a3636b f03212b955a7 rgw.default.rke-sh1-1.vylchc rke-sh1-1 *:8001 running (12d) 41s ago 22M 16.2.1 c757e4a3636b da486ce43fe5 rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 *:8000 running (12d) 2s ago 2y 16.2.1 c757e4a3636b ef4089d0aef2 rgw.default.rke-sh1-2.efkbum rke-sh1-2 *:8001 running (12d) 2s ago 2y 16.2.1 c757e4a3636b 9e053d5a2f7b rgw.default.rke-sh1-3.krfgey rke-sh1-3 *:8001 running (12d) 41s ago 9M 16.2.1 c757e4a3636b 45cd3d75edd3 rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 *:8000 running (12d) 41s ago 9M 16.2.1 c757e4a3636b e2710265a7f4 #tail -f /var/log/ceph/fcb373ce-7aaa-11eb-984f-e7c6e0038e87/ceph-mgr.rke-sh1-2.lxmguj.log 2024-03-06T09:24:42.468+0000 7fe68b500700 0 [dashboard DEBUG root] setting log level: INFO 2024-03-06T09:24:42.468+0000 7fe68b500700 1 mgr load Constructed class from module: dashboard 2024-03-06T09:24:42.468+0000 7fe68acff700 0 ms_deliver_dispatch: unhandled message 0x55f722292160 mon_map magic: 0 v1 from mon.0 v2:10.10.71.2:3300/0 2024-03-06T09:24:42.468+0000 7fe68b500700 0 [prometheus DEBUG root] setting log level based on debug_mgr: WARNING (1/5) 2024-03-06T09:24:42.468+0000 7fe68b500700 1 mgr load Constructed class from module: prometheus 2024-03-06T09:24:42.468+0000 7fe64110d700 0 [dashboard INFO root] server: ssl=no host=:: port=8082 2024-03-06T09:24:42.472+0000 7fe64110d700 0 [dashboard INFO root] Configured CherryPy, starting engine... 2024-03-06T09:24:42.472+0000 7fe64110d700 0 [dashboard INFO root] Starting engine... 2024-03-06T09:24:42.580+0000 7fe64110d700 0 [dashboard INFO root] Engine started... 2024-03-06T09:24:44.020+0000 7f0085fb8500 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-06T09:24:44.020+0000 7f0085fb8500 0 ceph version 16.2.2 (e8f22dde28889481f4dda2beb8a07788204821d3) pacific (stable), process ceph-mgr, pid 7 2024-03-06T09:24:44.020+0000 7f0085fb8500 0 pidfile_write: ignore empty --pid-file 2024-03-06T09:24:44.044+0000 7f0085fb8500 1 mgr[py] Loading python module 'alerts' 2024-03-06T09:24:44.156+0000 7f0085fb8500 1 mgr[py] Loading python module 'balancer' 2024-03-06T09:24:44.240+0000 7f0085fb8500 1 mgr[py] Loading python module 'cephadm' 2024-03-06T09:24:44.484+0000 7f0085fb8500 1 mgr[py] Loading python module 'crash' 2024-03-06T09:24:44.568+0000 7f0085fb8500 1 mgr[py] Loading python module 'dashboard' 2024-03-06T09:24:45.100+0000 7f0085fb8500 1 mgr[py] Loading python module 'devicehealth' 2024-03-06T09:24:45.184+0000 7f0085fb8500 1 mgr[py] Loading python module 'diskprediction_local' 2024-03-06T09:24:45.396+0000 7f0085fb8500 1 mgr[py] Loading python module 'influx' 2024-03-06T09:24:45.488+0000 7f0085fb8500 1 mgr[py] Loading python module 'insights' 2024-03-06T09:24:45.572+0000 7f0085fb8500 1 mgr[py] Loading python module 'iostat' 2024-03-06T09:24:45.724+0000 7f0085fb8500 1 mgr[py] Loading python module 'k8sevents' 2024-03-06T09:24:46.172+0000 7f0085fb8500 1 mgr[py] Loading python module 'localpool' 2024-03-06T09:24:46.260+0000 7f0085fb8500 1 mgr[py] Loading python module 'mds_autoscaler' 2024-03-06T09:24:46.416+0000 7f0085fb8500 1 mgr[py] Loading python module 'mirroring' 2024-03-06T09:24:46.528+0000 7f0085fb8500 1 mgr[py] Loading python module 'orchestrator' 2024-03-06T09:24:46.776+0000 7f0085fb8500 1 mgr[py] Loading python module 'osd_support' 2024-03-06T09:24:46.860+0000 7f0085fb8500 1 mgr[py] Loading python module 'pg_autoscaler' 2024-03-06T09:24:46.956+0000 7f0085fb8500 1 mgr[py] Loading python module 'progress' 2024-03-06T09:24:47.052+0000 7f0085fb8500 1 mgr[py] Loading python module 'prometheus' 2024-03-06T09:24:47.524+0000 7f0085fb8500 1 mgr[py] Loading python module 'rbd_support' 2024-03-06T09:24:47.640+0000 7f0085fb8500 1 mgr[py] Loading python module 'restful' 2024-03-06T09:24:47.924+0000 7f0085fb8500 1 mgr[py] Loading python module 'rook' 2024-03-06T09:24:48.536+0000 7f0085fb8500 1 mgr[py] Loading python module 'selftest' 2024-03-06T09:24:48.640+0000 7f0085fb8500 1 mgr[py] Loading python module 'snap_schedule' 2024-03-06T09:24:48.776+0000 7f0085fb8500 1 mgr[py] Loading python module 'stats' 2024-03-06T09:24:48.876+0000 7f0085fb8500 1 mgr[py] Loading python module 'status' 2024-03-06T09:24:48.984+0000 7f0085fb8500 1 mgr[py] Loading python module 'telegraf' 2024-03-06T09:24:49.088+0000 7f0085fb8500 1 mgr[py] Loading python module 'telemetry' 2024-03-06T09:24:49.248+0000 7f0085fb8500 1 mgr[py] Loading python module 'test_orchestrator' 2024-03-06T09:24:49.632+0000 7f0085fb8500 1 mgr[py] Loading python module 'volumes' 2024-03-06T09:24:49.832+0000 7f0085fb8500 1 mgr[py] Loading python module 'zabbix' 2024-03-06T09:24:49.936+0000 7f00739df700 0 [dashboard DEBUG root] setting log level: INFO 2024-03-06T09:24:49.936+0000 7f00739df700 1 mgr load Constructed class from module: dashboard 2024-03-06T09:24:49.936+0000 7f00731de700 0 ms_deliver_dispatch: unhandled message 0x556eb3224160 mon_map magic: 0 v1 from mon.2 v2:10.10.71.1:3300/0 2024-03-06T09:24:49.936+0000 7f00739df700 0 [prometheus DEBUG root] setting log level based on debug_mgr: WARNING (1/5) 2024-03-06T09:24:49.936+0000 7f00739df700 1 mgr load Constructed class from module: prometheus 2024-03-06T09:24:49.936+0000 7f00235e9700 0 [dashboard INFO root] server: ssl=no host=:: port=8082 2024-03-06T09:24:49.940+0000 7f00235e9700 0 [dashboard INFO root] Configured CherryPy, starting engine... 2024-03-06T09:24:49.940+0000 7f00235e9700 0 [dashboard INFO root] Starting engine... 2024-03-06T09:24:50.048+0000 7f00235e9700 0 [dashboard INFO root] Engine started... 2024-03-06T09:24:51.584+0000 7f0843ec9500 0 set uid:gid to 167:167 (ceph:ceph) 2024-03-06T09:24:51.584+0000 7f0843ec9500 0 ceph version 16.2.2 (e8f22dde28889481f4dda2beb8a07788204821d3) pacific (stable), process ceph-mgr, pid 7 2024-03-06T09:24:51.584+0000 7f0843ec9500 0 pidfile_write: ignore empty --pid-file # cephadm logs --fsid fcb373ce-7aaa-11eb-984f-e7c6e0038e87 --name mgr.rke-sh1-2.lxmguj Mar 06 09:27:18 rke-sh1-2 bash[623306]: [06/Mar/2024:09:27:18] ENGINE Bus STARTING Mar 06 09:27:18 rke-sh1-2 bash[623306]: CherryPy Checker: Mar 06 09:27:18 rke-sh1-2 bash[623306]: The Application mounted at '' has an empty config. Mar 06 09:27:18 rke-sh1-2 bash[623306]: [06/Mar/2024:09:27:18] ENGINE Serving on http://:::9283 Mar 06 09:27:18 rke-sh1-2 bash[623306]: [06/Mar/2024:09:27:18] ENGINE Bus STARTED Mar 06 09:27:18 rke-sh1-2 systemd[1]: Stopping Ceph mgr.rke-sh1-2.lxmguj for fcb373ce-7aaa-11eb-984f-e7c6e0038e87... Mar 06 09:27:18 rke-sh1-2 docker[624494]: ceph-fcb373ce-7aaa-11eb-984f-e7c6e0038e87-mgr.rke-sh1-2.lxmguj Mar 06 09:27:18 rke-sh1-2 systemd[1]: ceph-fcb373ce-7aaa-11eb-984f-e7c6e0038e87@xxxxxxx-sh1-2.lxmguj.service: Main process exited, code=exited, status=143/n/a Mar 06 09:27:18 rke-sh1-2 systemd[1]: ceph-fcb373ce-7aaa-11eb-984f-e7c6e0038e87@xxxxxxx-sh1-2.lxmguj.service: Failed with result 'exit-code'. Mar 06 09:27:18 rke-sh1-2 systemd[1]: Stopped Ceph mgr.rke-sh1-2.lxmguj for fcb373ce-7aaa-11eb-984f-e7c6e0038e87. Mar 06 09:27:19 rke-sh1-2 systemd[1]: Started Ceph mgr.rke-sh1-2.lxmguj for fcb373ce-7aaa-11eb-984f-e7c6e0038e87. The mgr.rke-sh1-2.lxmguj daemon is crashlooping. Do you have an idea on what going on ? Issue with the dashboard module ? Bets Regards, Edouard FAZENDA Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 From: Edouard FAZENDA Dear Ceph Community, I am in the process of upgrading ceph pacific 16.2.1 to 16.2.2 , I have followed the documentation : https://docs.ceph.com/en/pacific/cephadm/upgrade/ My cluster is in Healthy state , but the upgrade is not going forward , as on the cephadm logs I have the following : # Ceph -W cephadm 2024-03-06T08:39:11.653447+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Need to upgrade myself (mgr.rke-sh1-1.qskoyj) 2024-03-06T08:39:12.281386+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Updating mgr.rke-sh1-2.lxmguj 2024-03-06T08:39:12.286096+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 2024-03-06T08:39:19.347877+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-1: could not verify host allowed virtual ips 2024-03-06T08:39:19.347989+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-3: could not verify host allowed virtual ips 2024-03-06T08:39:19.366355+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Need to upgrade myself (mgr.rke-sh1-1.qskoyj) 2024-03-06T08:39:19.965822+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Updating mgr.rke-sh1-2.lxmguj 2024-03-06T08:39:19.969089+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 2024-03-06T08:39:26.961455+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-1: could not verify host allowed virtual ips 2024-03-06T08:39:26.961502+0000 mgr.rke-sh1-1.qskoyj [INF] Filtered out host rke-sh1-3: could not verify host allowed virtual ips 2024-03-06T08:39:26.973897+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Need to upgrade myself (mgr.rke-sh1-1.qskoyj) 2024-03-06T08:39:27.623773+0000 mgr.rke-sh1-1.qskoyj [INF] Upgrade: Updating mgr.rke-sh1-2.lxmguj 2024-03-06T08:39:27.628115+0000 mgr.rke-sh1-1.qskoyj [INF] Deploying daemon mgr.rke-sh1-2.lxmguj on rke-sh1-2 My public_network is set : root@rke-sh1-1:~# ceph config dump | grep public_network mon advanced public_network 10.10.71.0/24 * Do you have an idea why I have the following error : Filtered out host: could not verify host allowed virtual ips Current state of the upgrade : # ceph orch upgrade status { "target_image": "docker.io/ceph/ceph@sha256:8cdd8c7dfc7be5865255f0d59c048a1fb8d1335f6923996e2c2d9439499b5cf2", "in_progress": true, "services_complete": [], "progress": "0/35 ceph daemons upgraded", "message": "Currently upgrading mgr daemons" } progress: Upgrade to 16.2.2 (24m) [............................] Thanks for the help. Best Regards, Edouard FAZENDA Technical Support
Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 |
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
