Try the following command to fix the osd_remove_queue: ceph config-key set mgr/cephadm/osd_remove_queue [] After that, the orchestrator will be back. On Sat, Jan 25, 2025, 8:47 PM Devender Singh <devender@xxxxxxxxxx> wrote: > +Eugen > Lets follow “No recovery after removing node - > active+undersized+degraded-- removed osd using purge…”. Here. > > Sorry I missed ceph version which is 18.2.4. (with 5 nodes, 22osd each, > where I removed one node and all mess.) > > Regards > Dev > > > > > On Jan 25, 2025, at 11:34 AM, Devender Singh <devender@xxxxxxxxxx> > wrote: > > > > Hello Fredreic > > > > > > Thanks for your reply, Yes I also faced this issue after draining and > removing of the node. > > So used the same command and remove “original_weight” using ceph > config-key get mgr/cephadm/osd_remove_queue and injected file again. Which > resolved the orch issue. > > > > “Error ENOENT: Module not found - ceph orch commands stoppd working > > > > ceph config-key get mgr/cephadm/osd_remove_queue > osd_remove_queue.json > > > > Then only remove the "original_weight" key from that json and upload it > back to the config-key store: > > > > ceph config-key set mgr/cephadm/osd_remove_queue -i > osd_remove_queue_modified.json > > > > Then fail the mgr: > > > > ceph mgr fail” > > > > > > But now issue is, my cluster showing objects misplaced, whereas I had 5 > nodes with host failure domain with R3 pool (size 3 and min 2), EC with > 3+2. > > > > # ceph -s > > cluster: > > id: 384d7590-d018-11ee-b74c-5b2acfe0b35c > > health: HEALTH_WARN > > Degraded data redundancy: 2848547/29106793 objects degraded > (9.787%), 105 pgs degraded, 132 pgs undersized > > > > services: > > mon: 4 daemons, quorum node1,node5,node4,node2 (age 12h) > > mgr: node1.cvknae(active, since 12h), standbys: node4.foomun > > mds: 2/2 daemons up, 2 standby > > osd: 95 osds: 95 up (since 16h), 95 in (since 21h); 124 remapped pgs > > rgw: 2 daemons active (2 hosts, 1 zones) > > > > data: > > volumes: 2/2 healthy > > pools: 18 pools, 817 pgs > > objects: 6.06M objects, 20 TiB > > usage: 30 TiB used, 302 TiB / 332 TiB avail > > pgs: 2848547/29106793 objects degraded (9.787%) > > 2617833/29106793 objects misplaced (8.994%) > > 561 active+clean > > 124 active+clean+remapped > > 105 active+undersized+degraded > > 27 active+undersized > > > > io: > > client: 1.4 MiB/s rd, 4.0 MiB/s wr, 25 op/s rd, 545 op/s wr > > > > And when using 'ceph config-key ls’ it’s showing old node and osd’s. > > > > # ceph config-key ls|grep -i 03n > > "config-history/135/+osd/host:node3/osd_memory_target", > > "config-history/14990/+osd/host:node3/osd_memory_target", > > "config-history/14990/-osd/host:node3/osd_memory_target", > > "config-history/15003/+osd/host:node3/osd_memory_target", > > "config-history/15003/-osd/host:node3/osd_memory_target", > > "config-history/15016/+osd/host:node3/osd_memory_target", > > "config-history/15016/-osd/host:node3/osd_memory_target", > > "config-history/15017/+osd/host:node3/osd_memory_target", > > "config-history/15017/-osd/host:node3/osd_memory_target", > > "config-history/15022/+osd/host:node3/osd_memory_target", > > "config-history/15022/-osd/host:node3/osd_memory_target", > > "config-history/15024/+osd/host:node3/osd_memory_target", > > "config-history/15024/-osd/host:node3/osd_memory_target", > > "config-history/15025/+osd/host:node3/osd_memory_target", > > "config-history/15025/-osd/host:node3/osd_memory_target", > > "config-history/153/+osd/host:node3/osd_memory_target", > > "config-history/153/-osd/host:node3/osd_memory_target", > > "config-history/165/+mon.node3/container_image", > > "config-history/171/-mon.node3/container_image", > > "config-history/176/+client.crash.node3/container_image", > > "config-history/182/-client.crash.node3/container_image", > > "config-history/4276/+osd/host:node3/osd_memory_target", > > "config-history/4276/-osd/host:node3/osd_memory_target", > > "config-history/433/+client.ceph-exporter.node3/container_image", > > "config-history/439/-client.ceph-exporter.node3/container_image", > > "config-history/459/+osd/host:node3/osd_memory_target", > > "config-history/459/-osd/host:node3/osd_memory_target", > > "config-history/465/+osd/host:node3/osd_memory_target", > > "config-history/465/-osd/host:node3/osd_memory_target", > > "config-history/4867/+osd/host:node3/osd_memory_target", > > "config-history/4867/-osd/host:node3/osd_memory_target", > > "config-history/4878/+mon.node3/container_image", > > "config-history/4884/-mon.node3/container_image", > > "config-history/4889/+client.crash.node3/container_image", > > "config-history/4895/-client.crash.node3/container_image", > > > "config-history/5139/+mds.k8s-dev-cephfs.node3.iebxqn/container_image", > > > "config-history/5142/-mds.k8s-dev-cephfs.node3.iebxqn/container_image", > > "config-history/5150/+client.ceph-exporter.node3/container_image", > > "config-history/5156/-client.ceph-exporter.node3/container_image", > > "config-history/5179/+osd/host:node3/osd_memory_target", > > "config-history/5179/-osd/host:node3/osd_memory_target", > > "config-history/5183/+client.rgw.sea-dev.node3.betyqd/rgw_frontends", > > "config-history/5189/+osd/host:node3/osd_memory_target", > > "config-history/5189/-osd/host:node3/osd_memory_target", > > "config-history/6929/-client.rgw.sea-dev.node3.betyqd/rgw_frontends", > > "config-history/6933/+osd/host:node3/osd_memory_target", > > "config-history/6933/-osd/host:node3/osd_memory_target", > > "config-history/9710/+osd/host:node3/osd_memory_target", > > "config-history/9710/-osd/host:node3/osd_memory_target", > > "config/osd/host:node3/osd_memory_target”, > > > > > > Regards > > Dev > > > >> On Jan 25, 2025, at 4:39 AM, Frédéric Nass < > frederic.nass@xxxxxxxxxxxxxxxx> wrote: > >> > >> Hi, > >> > >> I've seen this happening on a test cluster after draining a host that > also had a MGR service. Can you check if Eugen's solution here [1] helps in > your case ? And maybe investigate 'ceph config-key ls' for any issues in > config keys ? > >> > >> Regards, > >> Frédéric. > >> > >> [1] https://www.spinics.net/lists/ceph-users/msg83667.html < > https://www.google.com/url?q=https://www.spinics.net/lists/ceph-users/msg83667.html&source=gmail-imap&ust=1738413580000000&usg=AOvVaw3Zk70LrQ6SrLX02gJ7Cowl > > > >> > >> De : Devender Singh <devender@xxxxxxxxxx> > >> Envoyé : samedi 25 janvier 2025 06:27 > >> À : Fnu Virender Kumar > >> Cc: ceph-users > >> Objet : Re: Error ENOENT: Module not found > >> > >> Thanks for you reply… but those command not working as its an always > module..but strange still showing error, > >> > >> # ceph mgr module enable orchestrator > >> module 'orchestrator' is already enabled (always-on) > >> > >> # ceph orch set backend — returns successfully… > >> > >> # # ceph orch ls > >> Error ENOENT: No orchestrator configured (try `ceph orch set backend`) > >> > >> Its revolving between same error.. > >> > >> Root Cause: I removed a hosts and its odd’s and after some time above > error started automatically. > >> > >> Earlier in the had 5 nodes but now 4.. Cluster is showing unclean pg > but not doing anything.. > >> > >> But big error is Error ENOENT: > >> > >> > >> Regards > >> Dev > >> > >> > On Jan 24, 2025, at 4:59 PM, Fnu Virender Kumar < > virenderk@xxxxxxxxxxxx> wrote: > >> > > >> > Did you try > >> > > >> > Ceph mgr module enable orchestrator > >> > Ceph orch set backend > >> > Ceph orch ls > >> > > >> > Check the mgr service daemon as well > >> > Ceph -s > >> > > >> > > >> > Regards > >> > Virender > >> > From: Devender Singh <devender@xxxxxxxxxx> > >> > Sent: Friday, January 24, 2025 6:34:43 PM > >> > To: ceph-users <ceph-users@xxxxxxx> > >> > Subject: Error ENOENT: Module not found > >> > > >> > > >> > Hello all > >> > > >> > Any quick fix for … > >> > > >> > root@sea-devnode1:~# ceph orch ls > >> > Error ENOENT: Module not found > >> > > >> > > >> > Regards > >> > Dev > >> > _______________________________________________ > >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |