I don't think the number of mons should have any effect on this. Looking at your logs, the interesting thing is that all the messages are so close together. Was this before having stopped the upgrade? On Fri, Sep 2, 2022 at 2:53 PM Satish Patel <satish.txt@xxxxxxxxx> wrote: > Do you think this is because I have only a single MON daemon running? I > have only two nodes. > > On Fri, Sep 2, 2022 at 2:39 PM Satish Patel <satish.txt@xxxxxxxxx> wrote: > >> Adam, >> >> I have enabled debug and my logs flood with the following. I am going to >> try some stuff from your provided mailing list and see.. >> >> root@ceph1:~# tail -f >> /var/log/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/ceph.cephadm.log >> 2022-09-02T18:38:21.754391+0000 mgr.ceph2.huidoh (mgr.344392) 211198 : >> cephadm [DBG] 0 OSDs are scheduled for removal: [] >> 2022-09-02T18:38:21.754519+0000 mgr.ceph2.huidoh (mgr.344392) 211199 : >> cephadm [DBG] Saving [] to store >> 2022-09-02T18:38:21.757155+0000 mgr.ceph2.huidoh (mgr.344392) 211200 : >> cephadm [DBG] refreshing hosts and daemons >> 2022-09-02T18:38:21.758065+0000 mgr.ceph2.huidoh (mgr.344392) 211201 : >> cephadm [DBG] _check_for_strays >> 2022-09-02T18:38:21.758334+0000 mgr.ceph2.huidoh (mgr.344392) 211202 : >> cephadm [DBG] 0 OSDs are scheduled for removal: [] >> 2022-09-02T18:38:21.758455+0000 mgr.ceph2.huidoh (mgr.344392) 211203 : >> cephadm [DBG] Saving [] to store >> 2022-09-02T18:38:21.761001+0000 mgr.ceph2.huidoh (mgr.344392) 211204 : >> cephadm [DBG] refreshing hosts and daemons >> 2022-09-02T18:38:21.762092+0000 mgr.ceph2.huidoh (mgr.344392) 211205 : >> cephadm [DBG] _check_for_strays >> 2022-09-02T18:38:21.762357+0000 mgr.ceph2.huidoh (mgr.344392) 211206 : >> cephadm [DBG] 0 OSDs are scheduled for removal: [] >> 2022-09-02T18:38:21.762480+0000 mgr.ceph2.huidoh (mgr.344392) 211207 : >> cephadm [DBG] Saving [] to store >> >> On Fri, Sep 2, 2022 at 12:17 PM Adam King <adking@xxxxxxxxxx> wrote: >> >>> hmm, okay. It seems like cephadm is stuck in general rather than an >>> issue specific to the upgrade. I'd first make sure the orchestrator isn't >>> paused (just running "ceph orch resume" should be enough, it's idempotent). >>> >>> Beyond that, there was someone else who had an issue with things getting >>> stuck that was resolved in this thread >>> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/NKKLV5TMHFA3ERGCMJ3M7BVLA5PGIR4M/#NKKLV5TMHFA3ERGCMJ3M7BVLA5PGIR4M >>> <https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/NKKLV5TMHFA3ERGCMJ3M7BVLA5PGIR4M/#NKKLV5TMHFA3ERGCMJ3M7BVLA5PGIR4M> that >>> might be worth a look. >>> >>> If you haven't already, it's possible stopping the upgrade is a good >>> idea, as maybe that's interfering with it getting to the point where it >>> does the redeploy. >>> >>> If none of those help, it might be worth setting the log level to debug >>> and seeing where things are ending up ("ceph config set mgr >>> mgr/cephadm/log_to_cluster_level debug; ceph orch ps --refresh" then >>> waiting a few minutes before running "ceph log last 100 debug cephadm" (not >>> 100% on format of that command, if it fails try just "ceph log last >>> cephadm"). We could maybe get more info on why it's not performing the >>> redeploy from those debug logs. Just remember to set the log level back >>> after 'ceph config set mgr mgr/cephadm/log_to_cluster_level info' as debug >>> logs are quite verbose. >>> >>> On Fri, Sep 2, 2022 at 11:39 AM Satish Patel <satish.txt@xxxxxxxxx> >>> wrote: >>> >>>> Hi Adam, >>>> >>>> As you said, i did following >>>> >>>> $ ceph orch daemon redeploy mgr.ceph1.smfvfd >>>> quay.io/ceph/ceph:v16.2.10 >>>> >>>> Noticed following line in logs but then no activity nothing, still >>>> standby mgr running in older version >>>> >>>> 2022-09-02T15:35:45.753093+0000 mgr.ceph2.huidoh (mgr.344392) 2226 : >>>> cephadm [INF] Schedule redeploy daemon mgr.ceph1.smfvfd >>>> 2022-09-02T15:36:17.279190+0000 mgr.ceph2.huidoh (mgr.344392) 2245 : >>>> cephadm [INF] refreshing ceph2 facts >>>> 2022-09-02T15:36:17.984478+0000 mgr.ceph2.huidoh (mgr.344392) 2246 : >>>> cephadm [INF] refreshing ceph1 facts >>>> 2022-09-02T15:37:17.663730+0000 mgr.ceph2.huidoh (mgr.344392) 2284 : >>>> cephadm [INF] refreshing ceph2 facts >>>> 2022-09-02T15:37:18.386586+0000 mgr.ceph2.huidoh (mgr.344392) 2285 : >>>> cephadm [INF] refreshing ceph1 facts >>>> >>>> I am not seeing any image get downloaded also >>>> >>>> root@ceph1:~# docker image ls >>>> REPOSITORY TAG IMAGE ID CREATED >>>> SIZE >>>> quay.io/ceph/ceph v15 93146564743f 3 weeks >>>> ago 1.2GB >>>> quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 4 months >>>> ago 558MB >>>> quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 6 months >>>> ago 204MB >>>> quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 12 months >>>> ago 57.5MB >>>> quay.io/ceph/ceph-grafana 6.7.4 557c83e11646 13 months >>>> ago 486MB >>>> quay.io/prometheus/prometheus v2.18.1 de242295e225 2 years >>>> ago 140MB >>>> quay.io/prometheus/alertmanager v0.20.0 0881eb8f169f 2 years >>>> ago 52.1MB >>>> quay.io/prometheus/node-exporter v0.18.1 e5a616e4b9cf 3 years >>>> ago 22.9MB >>>> >>>> >>>> On Fri, Sep 2, 2022 at 11:06 AM Adam King <adking@xxxxxxxxxx> wrote: >>>> >>>>> hmm, at this point, maybe we should just try manually upgrading the >>>>> mgr daemons and then move from there. First, just stop the upgrade "ceph >>>>> orch upgrade stop". If you figure out which of the two mgr daemons is the >>>>> standby (it should say which one is active in "ceph -s" output) and then do >>>>> a "ceph orch daemon redeploy <standby-mgr-name> >>>>> quay.io/ceph/ceph:v16.2.10" it should redeploy that specific mgr with >>>>> the new version. You could then do a "ceph mgr fail" to swap which of the >>>>> mgr daemons is active, then do another "ceph orch daemon redeploy >>>>> <standby-mgr-name> quay.io/ceph/ceph:v16.2.10" where the standby is >>>>> now the other mgr still on 15.2.17. Once the mgr daemons are both upgraded >>>>> to the new version, run a "ceph orch redeploy mgr" and then "ceph orch >>>>> upgrade start --image quay.io/ceph/ceph:v16.2.10" and see if it goes >>>>> better. >>>>> >>>>> On Fri, Sep 2, 2022 at 10:36 AM Satish Patel <satish.txt@xxxxxxxxx> >>>>> wrote: >>>>> >>>>>> Hi Adam, >>>>>> >>>>>> I run the following command to upgrade but it looks like nothing is >>>>>> happening >>>>>> >>>>>> $ ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.10 >>>>>> >>>>>> Status message is empty.. >>>>>> >>>>>> root@ceph1:~# ceph orch upgrade status >>>>>> { >>>>>> "target_image": "quay.io/ceph/ceph:v16.2.10", >>>>>> "in_progress": true, >>>>>> "services_complete": [], >>>>>> "message": "" >>>>>> } >>>>>> >>>>>> Nothing in Logs >>>>>> >>>>>> root@ceph1:~# tail -f >>>>>> /var/log/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/ceph.cephadm.log >>>>>> 2022-09-02T14:31:52.597661+0000 mgr.ceph2.huidoh (mgr.344392) 174 : >>>>>> cephadm [INF] refreshing ceph2 facts >>>>>> 2022-09-02T14:31:52.991450+0000 mgr.ceph2.huidoh (mgr.344392) 176 : >>>>>> cephadm [INF] refreshing ceph1 facts >>>>>> 2022-09-02T14:32:52.965092+0000 mgr.ceph2.huidoh (mgr.344392) 207 : >>>>>> cephadm [INF] refreshing ceph2 facts >>>>>> 2022-09-02T14:32:53.369789+0000 mgr.ceph2.huidoh (mgr.344392) 208 : >>>>>> cephadm [INF] refreshing ceph1 facts >>>>>> 2022-09-02T14:33:53.367986+0000 mgr.ceph2.huidoh (mgr.344392) 239 : >>>>>> cephadm [INF] refreshing ceph2 facts >>>>>> 2022-09-02T14:33:53.760427+0000 mgr.ceph2.huidoh (mgr.344392) 240 : >>>>>> cephadm [INF] refreshing ceph1 facts >>>>>> 2022-09-02T14:34:53.754277+0000 mgr.ceph2.huidoh (mgr.344392) 272 : >>>>>> cephadm [INF] refreshing ceph2 facts >>>>>> 2022-09-02T14:34:54.162503+0000 mgr.ceph2.huidoh (mgr.344392) 273 : >>>>>> cephadm [INF] refreshing ceph1 facts >>>>>> 2022-09-02T14:35:54.133467+0000 mgr.ceph2.huidoh (mgr.344392) 305 : >>>>>> cephadm [INF] refreshing ceph2 facts >>>>>> 2022-09-02T14:35:54.522171+0000 mgr.ceph2.huidoh (mgr.344392) 306 : >>>>>> cephadm [INF] refreshing ceph1 facts >>>>>> >>>>>> In progress that mesg stuck there for long time >>>>>> >>>>>> root@ceph1:~# ceph -s >>>>>> cluster: >>>>>> id: f270ad9e-1f6f-11ed-b6f8-a539d87379ea >>>>>> health: HEALTH_OK >>>>>> >>>>>> services: >>>>>> mon: 1 daemons, quorum ceph1 (age 9h) >>>>>> mgr: ceph2.huidoh(active, since 9m), standbys: ceph1.smfvfd >>>>>> osd: 4 osds: 4 up (since 9h), 4 in (since 11h) >>>>>> >>>>>> data: >>>>>> pools: 5 pools, 129 pgs >>>>>> objects: 20.06k objects, 83 GiB >>>>>> usage: 168 GiB used, 632 GiB / 800 GiB avail >>>>>> pgs: 129 active+clean >>>>>> >>>>>> io: >>>>>> client: 12 KiB/s wr, 0 op/s rd, 1 op/s wr >>>>>> >>>>>> progress: >>>>>> Upgrade to quay.io/ceph/ceph:v16.2.10 (0s) >>>>>> [............................] >>>>>> >>>>>> On Fri, Sep 2, 2022 at 10:25 AM Satish Patel <satish.txt@xxxxxxxxx> >>>>>> wrote: >>>>>> >>>>>>> It Looks like I did it with the following command. >>>>>>> >>>>>>> $ ceph orch daemon add mgr ceph2:10.73.0.192 >>>>>>> >>>>>>> Now i can see two with same version 15.x >>>>>>> >>>>>>> root@ceph1:~# ceph orch ps --daemon-type mgr >>>>>>> NAME HOST STATUS REFRESHED AGE VERSION >>>>>>> IMAGE NAME >>>>>>> IMAGE ID CONTAINER ID >>>>>>> mgr.ceph1.smfvfd ceph1 running (8h) 41s ago 8h 15.2.17 >>>>>>> quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca >>>>>>> 93146564743f 1aab837306d2 >>>>>>> mgr.ceph2.huidoh ceph2 running (60s) 110s ago 60s 15.2.17 >>>>>>> quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca >>>>>>> 93146564743f 294fd6ab6c97 >>>>>>> >>>>>>> On Fri, Sep 2, 2022 at 10:19 AM Satish Patel <satish.txt@xxxxxxxxx> >>>>>>> wrote: >>>>>>> >>>>>>>> Let's come back to the original question: how to bring back the >>>>>>>> second mgr? >>>>>>>> >>>>>>>> root@ceph1:~# ceph orch apply mgr 2 >>>>>>>> Scheduled mgr update... >>>>>>>> >>>>>>>> Nothing happened with above command, logs saying nothing >>>>>>>> >>>>>>>> 2022-09-02T14:16:20.407927+0000 mgr.ceph1.smfvfd (mgr.334626) 16939 >>>>>>>> : cephadm [INF] refreshing ceph2 facts >>>>>>>> 2022-09-02T14:16:40.247195+0000 mgr.ceph1.smfvfd (mgr.334626) 16952 >>>>>>>> : cephadm [INF] Saving service mgr spec with placement count:2 >>>>>>>> 2022-09-02T14:16:53.106919+0000 mgr.ceph1.smfvfd (mgr.334626) 16961 >>>>>>>> : cephadm [INF] Saving service mgr spec with placement count:2 >>>>>>>> 2022-09-02T14:17:19.135203+0000 mgr.ceph1.smfvfd (mgr.334626) 16975 >>>>>>>> : cephadm [INF] refreshing ceph1 facts >>>>>>>> 2022-09-02T14:17:20.780496+0000 mgr.ceph1.smfvfd (mgr.334626) 16977 >>>>>>>> : cephadm [INF] refreshing ceph2 facts >>>>>>>> 2022-09-02T14:18:19.502034+0000 mgr.ceph1.smfvfd (mgr.334626) 17008 >>>>>>>> : cephadm [INF] refreshing ceph1 facts >>>>>>>> 2022-09-02T14:18:21.127973+0000 mgr.ceph1.smfvfd (mgr.334626) 17010 >>>>>>>> : cephadm [INF] refreshing ceph2 facts >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Sep 2, 2022 at 10:15 AM Satish Patel <satish.txt@xxxxxxxxx> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Adam, >>>>>>>>> >>>>>>>>> Wait..wait.. now it's working suddenly without doing anything.. >>>>>>>>> very odd >>>>>>>>> >>>>>>>>> root@ceph1:~# ceph orch ls >>>>>>>>> NAME RUNNING REFRESHED AGE PLACEMENT IMAGE >>>>>>>>> NAME >>>>>>>>> IMAGE ID >>>>>>>>> alertmanager 1/1 5s ago 2w count:1 >>>>>>>>> quay.io/prometheus/alertmanager:v0.20.0 >>>>>>>>> 0881eb8f169f >>>>>>>>> crash 2/2 5s ago 2w * >>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>> 93146564743f >>>>>>>>> grafana 1/1 5s ago 2w count:1 >>>>>>>>> quay.io/ceph/ceph-grafana:6.7.4 >>>>>>>>> 557c83e11646 >>>>>>>>> mgr 1/2 5s ago 8h count:2 >>>>>>>>> quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca >>>>>>>>> 93146564743f >>>>>>>>> mon 1/2 5s ago 8h ceph1;ceph2 >>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>> 93146564743f >>>>>>>>> node-exporter 2/2 5s ago 2w * >>>>>>>>> quay.io/prometheus/node-exporter:v0.18.1 >>>>>>>>> e5a616e4b9cf >>>>>>>>> osd.osd_spec_default 4/0 5s ago - <unmanaged> >>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>> 93146564743f >>>>>>>>> prometheus 1/1 5s ago 2w count:1 >>>>>>>>> quay.io/prometheus/prometheus:v2.18.1 >>>>>>>>> >>>>>>>>> On Fri, Sep 2, 2022 at 10:13 AM Satish Patel <satish.txt@xxxxxxxxx> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I can see that in the output but I'm not sure how to get rid of >>>>>>>>>> it. >>>>>>>>>> >>>>>>>>>> root@ceph1:~# ceph orch ps --refresh >>>>>>>>>> NAME >>>>>>>>>> HOST STATUS REFRESHED AGE VERSION IMAGE NAME >>>>>>>>>> >>>>>>>>>> IMAGE ID CONTAINER ID >>>>>>>>>> alertmanager.ceph1 >>>>>>>>>> ceph1 running (9h) 64s ago 2w 0.20.0 >>>>>>>>>> quay.io/prometheus/alertmanager:v0.20.0 >>>>>>>>>> 0881eb8f169f ba804b555378 >>>>>>>>>> cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d >>>>>>>>>> ceph2 stopped 65s ago - <unknown> <unknown> >>>>>>>>>> <unknown> >>>>>>>>>> <unknown> >>>>>>>>>> crash.ceph1 >>>>>>>>>> ceph1 running (9h) 64s ago 2w 15.2.17 >>>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>>> 93146564743f a3a431d834fc >>>>>>>>>> crash.ceph2 >>>>>>>>>> ceph2 running (9h) 65s ago 13d 15.2.17 >>>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>>> 93146564743f 3c963693ff2b >>>>>>>>>> grafana.ceph1 >>>>>>>>>> ceph1 running (9h) 64s ago 2w 6.7.4 >>>>>>>>>> quay.io/ceph/ceph-grafana:6.7.4 >>>>>>>>>> 557c83e11646 7583a8dc4c61 >>>>>>>>>> mgr.ceph1.smfvfd >>>>>>>>>> ceph1 running (8h) 64s ago 8h 15.2.17 >>>>>>>>>> quay.io/ceph/ceph@sha256:c08064dde4bba4e72a1f55d90ca32df9ef5aafab82efe2e0a0722444a5aaacca >>>>>>>>>> 93146564743f 1aab837306d2 >>>>>>>>>> mon.ceph1 >>>>>>>>>> ceph1 running (9h) 64s ago 2w 15.2.17 >>>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>>> 93146564743f c1d155d8c7ad >>>>>>>>>> node-exporter.ceph1 >>>>>>>>>> ceph1 running (9h) 64s ago 2w 0.18.1 >>>>>>>>>> quay.io/prometheus/node-exporter:v0.18.1 >>>>>>>>>> e5a616e4b9cf 2ff235fe0e42 >>>>>>>>>> node-exporter.ceph2 >>>>>>>>>> ceph2 running (9h) 65s ago 13d 0.18.1 >>>>>>>>>> quay.io/prometheus/node-exporter:v0.18.1 >>>>>>>>>> e5a616e4b9cf 17678b9ba602 >>>>>>>>>> osd.0 >>>>>>>>>> ceph1 running (9h) 64s ago 13d 15.2.17 >>>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>>> 93146564743f d0fd73b777a3 >>>>>>>>>> osd.1 >>>>>>>>>> ceph1 running (9h) 64s ago 13d 15.2.17 >>>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>>> 93146564743f 049120e83102 >>>>>>>>>> osd.2 >>>>>>>>>> ceph2 running (9h) 65s ago 13d 15.2.17 >>>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>>> 93146564743f 8700e8cefd1f >>>>>>>>>> osd.3 >>>>>>>>>> ceph2 running (9h) 65s ago 13d 15.2.17 >>>>>>>>>> quay.io/ceph/ceph:v15 >>>>>>>>>> 93146564743f 9c71bc87ed16 >>>>>>>>>> prometheus.ceph1 >>>>>>>>>> ceph1 running (9h) 64s ago 2w 2.18.1 >>>>>>>>>> quay.io/prometheus/prometheus:v2.18.1 >>>>>>>>>> de242295e225 74a538efd61e >>>>>>>>>> >>>>>>>>>> On Fri, Sep 2, 2022 at 10:10 AM Adam King <adking@xxxxxxxxxx> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> maybe also a "ceph orch ps --refresh"? It might still have the >>>>>>>>>>> old cached daemon inventory from before you remove the files. >>>>>>>>>>> >>>>>>>>>>> On Fri, Sep 2, 2022 at 9:57 AM Satish Patel < >>>>>>>>>>> satish.txt@xxxxxxxxx> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Adam, >>>>>>>>>>>> >>>>>>>>>>>> I have deleted file located here - rm >>>>>>>>>>>> /var/lib/ceph/f270ad9e-1f6f-11ed-b6f8-a539d87379ea/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d >>>>>>>>>>>> >>>>>>>>>>>> But still getting the same error, do i need to do anything >>>>>>>>>>>> else? >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Sep 2, 2022 at 9:51 AM Adam King <adking@xxxxxxxxxx> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Okay, I'm wondering if this is an issue with version mismatch. >>>>>>>>>>>>> Having previously had a 16.2.10 mgr and then now having a 15.2.17 one that >>>>>>>>>>>>> doesn't expect this sort of thing to be present. Either way, I'd think just >>>>>>>>>>>>> deleting this cephadm.7ce656a8721deb5054c37b0cfb9038 >>>>>>>>>>>>> 1522d521dde51fb0c5a2142314d663f63d (and any others like it) file >>>>>>>>>>>>> would be the way forward to get orch ls working again. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Sep 2, 2022 at 9:44 AM Satish Patel < >>>>>>>>>>>>> satish.txt@xxxxxxxxx> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Adam, >>>>>>>>>>>>>> >>>>>>>>>>>>>> In cephadm ls i found the following service but i believe it >>>>>>>>>>>>>> was there before also. >>>>>>>>>>>>>> >>>>>>>>>>>>>> { >>>>>>>>>>>>>> "style": "cephadm:v1", >>>>>>>>>>>>>> "name": >>>>>>>>>>>>>> "cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", >>>>>>>>>>>>>> "fsid": "f270ad9e-1f6f-11ed-b6f8-a539d87379ea", >>>>>>>>>>>>>> "systemd_unit": >>>>>>>>>>>>>> "ceph-f270ad9e-1f6f-11ed-b6f8-a539d87379ea@cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d >>>>>>>>>>>>>> ", >>>>>>>>>>>>>> "enabled": false, >>>>>>>>>>>>>> "state": "stopped", >>>>>>>>>>>>>> "container_id": null, >>>>>>>>>>>>>> "container_image_name": null, >>>>>>>>>>>>>> "container_image_id": null, >>>>>>>>>>>>>> "version": null, >>>>>>>>>>>>>> "started": null, >>>>>>>>>>>>>> "created": null, >>>>>>>>>>>>>> "deployed": null, >>>>>>>>>>>>>> "configured": null >>>>>>>>>>>>>> }, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Look like remove didn't work >>>>>>>>>>>>>> >>>>>>>>>>>>>> root@ceph1:~# ceph orch rm cephadm >>>>>>>>>>>>>> Failed to remove service. <cephadm> was not found. >>>>>>>>>>>>>> >>>>>>>>>>>>>> root@ceph1:~# ceph orch rm >>>>>>>>>>>>>> cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d >>>>>>>>>>>>>> Failed to remove service. >>>>>>>>>>>>>> <cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d> >>>>>>>>>>>>>> was not found. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Sep 2, 2022 at 8:27 AM Adam King <adking@xxxxxxxxxx> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> this looks like an old traceback you would get if you ended >>>>>>>>>>>>>>> up with a service type that shouldn't be there somehow. The things I'd >>>>>>>>>>>>>>> probably check are that "cephadm ls" on either host definitely doesn't >>>>>>>>>>>>>>> report and strange things that aren't actually daemons in your cluster such >>>>>>>>>>>>>>> as "cephadm.<hash>". Another thing you could maybe try, as I believe the >>>>>>>>>>>>>>> assertion it's giving is for an unknown service type here ("AssertionError: >>>>>>>>>>>>>>> cephadm"), is just "ceph orch rm cephadm" which would maybe cause it to >>>>>>>>>>>>>>> remove whatever it thinks is this "cephadm" service that it has deployed. >>>>>>>>>>>>>>> Lastly, you could try having the mgr you manually deploy be a 16.2.10 one >>>>>>>>>>>>>>> instead of 15.2.17 (I'm assuming here, but the line numbers in that >>>>>>>>>>>>>>> traceback suggest octopus). The 16.2.10 one is just much less likely to >>>>>>>>>>>>>>> have a bug that causes something like this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Sep 2, 2022 at 1:41 AM Satish Patel < >>>>>>>>>>>>>>> satish.txt@xxxxxxxxx> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Now when I run "ceph orch ps" it works but the following >>>>>>>>>>>>>>>> command throws an >>>>>>>>>>>>>>>> error. Trying to bring up second mgr using ceph orch apply >>>>>>>>>>>>>>>> mgr command but >>>>>>>>>>>>>>>> didn't help >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> root@ceph1:/ceph-disk# ceph version >>>>>>>>>>>>>>>> ceph version 15.2.17 >>>>>>>>>>>>>>>> (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus >>>>>>>>>>>>>>>> (stable) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> root@ceph1:/ceph-disk# ceph orch ls >>>>>>>>>>>>>>>> Error EINVAL: Traceback (most recent call last): >>>>>>>>>>>>>>>> File "/usr/share/ceph/mgr/mgr_module.py", line 1212, in >>>>>>>>>>>>>>>> _handle_command >>>>>>>>>>>>>>>> return self.handle_command(inbuf, cmd) >>>>>>>>>>>>>>>> File "/usr/share/ceph/mgr/orchestrator/_interface.py", >>>>>>>>>>>>>>>> line 140, in >>>>>>>>>>>>>>>> handle_command >>>>>>>>>>>>>>>> return dispatch[cmd['prefix']].call(self, cmd, inbuf) >>>>>>>>>>>>>>>> File "/usr/share/ceph/mgr/mgr_module.py", line 320, in >>>>>>>>>>>>>>>> call >>>>>>>>>>>>>>>> return self.func(mgr, **kwargs) >>>>>>>>>>>>>>>> File "/usr/share/ceph/mgr/orchestrator/_interface.py", >>>>>>>>>>>>>>>> line 102, in >>>>>>>>>>>>>>>> <lambda> >>>>>>>>>>>>>>>> wrapper_copy = lambda *l_args, **l_kwargs: >>>>>>>>>>>>>>>> wrapper(*l_args, **l_kwargs) >>>>>>>>>>>>>>>> File "/usr/share/ceph/mgr/orchestrator/_interface.py", >>>>>>>>>>>>>>>> line 91, in wrapper >>>>>>>>>>>>>>>> return func(*args, **kwargs) >>>>>>>>>>>>>>>> File "/usr/share/ceph/mgr/orchestrator/module.py", line >>>>>>>>>>>>>>>> 503, in >>>>>>>>>>>>>>>> _list_services >>>>>>>>>>>>>>>> raise_if_exception(completion) >>>>>>>>>>>>>>>> File "/usr/share/ceph/mgr/orchestrator/_interface.py", >>>>>>>>>>>>>>>> line 642, in >>>>>>>>>>>>>>>> raise_if_exception >>>>>>>>>>>>>>>> raise e >>>>>>>>>>>>>>>> AssertionError: cephadm >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Sep 2, 2022 at 1:32 AM Satish Patel < >>>>>>>>>>>>>>>> satish.txt@xxxxxxxxx> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> > nevermind, i found doc related that and i am able to get >>>>>>>>>>>>>>>> 1 mgr up - >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-a-mgr-daemon >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > On Fri, Sep 2, 2022 at 1:21 AM Satish Patel < >>>>>>>>>>>>>>>> satish.txt@xxxxxxxxx> wrote: >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >> Folks, >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> I am having little fun time with cephadm and it's very >>>>>>>>>>>>>>>> annoying to deal >>>>>>>>>>>>>>>> >> with it >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> I have deployed a ceph cluster using cephadm on two >>>>>>>>>>>>>>>> nodes. Now when i was >>>>>>>>>>>>>>>> >> trying to upgrade and noticed hiccups where it just >>>>>>>>>>>>>>>> upgraded a single mgr >>>>>>>>>>>>>>>> >> with 16.2.10 but not other so i started messing around >>>>>>>>>>>>>>>> and somehow I >>>>>>>>>>>>>>>> >> deleted both mgr in the thought that cephadm will >>>>>>>>>>>>>>>> recreate them. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> Now i don't have any single mgr so my ceph orch command >>>>>>>>>>>>>>>> hangs forever and >>>>>>>>>>>>>>>> >> looks like a chicken egg issue. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> How do I recover from this? If I can't run the ceph orch >>>>>>>>>>>>>>>> command, I won't >>>>>>>>>>>>>>>> >> be able to redeploy my mgr daemons. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> I am not able to find any mgr in the following command >>>>>>>>>>>>>>>> on both nodes. >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> $ cephadm ls | grep mgr >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>>>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |