with the log to cluster level already on debug, if you do a "ceph mgr fail"
what does cephadm log to the cluster before it reports sleeping? It should
at least be doing something if it's responsive at all. Also, in "ceph orch
ps" and "ceph orch device ls" are the REFRESHED columns reporting that
they've refreshed the info recently (last 10 minutes for daemons, last 30
minutes for devices)?
On Tue, Aug 15, 2023 at 3:46 AM Robert Sander <r.sander@xxxxxxxxxxxxxxxxxxx>
wrote:
> Hi,
>
> A healthy 16.2.7 cluster should get an upgrade to 16.2.13.
>
> ceph orch upgrade start --ceph-version 16.2.13
>
> did upgrade MONs, MGRs and 25% of the OSDs and is now stuck.
>
> We tried several "ceph orch upgrade stop" and starts again.
> We "failed" the active MGR but no progress.
> We set the debug logging with "ceph config set mgr
> mgr/cephadm/log_to_cluster_level debug" but it only tells that it starts:
>
> 2023-08-15T09:05:58.548896+0200 mgr.cephmon01 [INF] Upgrade: Started with
> target quay.io/ceph/ceph:v16.2.13
>
> How can we check what is happening (or not happening) here?
> How do we get cephadm to complete the task?
>
> Current status is:
>
> # ceph orch upgrade status
> {
> "target_image": "quay.io/ceph/ceph:v16.2.13",
> "in_progress": true,
> "which": "Upgrading all daemon types on all hosts",
> "services_complete": [],
> "progress": "",
> "message": "",
> "is_paused": false
> }
>
> # ceph -s
> cluster:
> id: 3098199a-c7f5-4baf-901c-f178131be6f4
> health: HEALTH_WARN
> There are daemons running an older version of ceph
>
> services:
> mon: 5 daemons, quorum
> cephmon02,cephmon01,cephmon03,cephmon04,cephmon05 (age 4d)
> mgr: cephmon03(active, since 8d), standbys: cephmon01, cephmon02
> mds: 2/2 daemons up, 1 standby, 2 hot standby
> osd: 202 osds: 202 up (since 11d), 202 in (since 13d)
> rgw: 2 daemons active (2 hosts, 1 zones)
>
> data:
> volumes: 2/2 healthy
> pools: 11 pools, 4961 pgs
> objects: 98.84M objects, 347 TiB
> usage: 988 TiB used, 1.3 PiB / 2.3 PiB avail
> pgs: 4942 active+clean
> 19 active+clean+scrubbing+deep
>
> io:
> client: 89 MiB/s rd, 598 MiB/s wr, 25 op/s rd, 157 op/s wr
>
> progress:
> Upgrade to quay.io/ceph/ceph:v16.2.13 (0s)
> [............................]
>
> # ceph versions
> {
> "mon": {
> "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e)
> pacific (stable)": 5
> },
> "mgr": {
> "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e)
> pacific (stable)": 3
> },
> "osd": {
> "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e)
> pacific (stable)": 48,
> "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503)
> pacific (stable)": 154
> },
> "mds": {
> "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503)
> pacific (stable)": 5
> },
> "rgw": {
> "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503)
> pacific (stable)": 2
> },
> "overall": {
> "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e)
> pacific (stable)": 56,
> "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503)
> pacific (stable)": 161
> }
> }
>
> Regards
> --
> Robert Sander
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> https://www.heinlein-support.de
>
> Tel: 030 / 405051-43
> Fax: 030 / 405051-19
>
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
