+Eugen Lets follow “No recovery after removing node - active+undersized+degraded-- removed osd using purge…”. Here. Sorry I missed ceph version which is 18.2.4. (with 5 nodes, 22osd each, where I removed one node and all mess.) Regards Dev > On Jan 25, 2025, at 11:34 AM, Devender Singh <devender@xxxxxxxxxx> wrote: > > Hello Fredreic > > > Thanks for your reply, Yes I also faced this issue after draining and removing of the node. > So used the same command and remove “original_weight” using ceph config-key get mgr/cephadm/osd_remove_queue and injected file again. Which resolved the orch issue. > > “Error ENOENT: Module not found - ceph orch commands stoppd working > > ceph config-key get mgr/cephadm/osd_remove_queue > osd_remove_queue.json > > Then only remove the "original_weight" key from that json and upload it back to the config-key store: > > ceph config-key set mgr/cephadm/osd_remove_queue -i osd_remove_queue_modified.json > > Then fail the mgr: > > ceph mgr fail” > > > But now issue is, my cluster showing objects misplaced, whereas I had 5 nodes with host failure domain with R3 pool (size 3 and min 2), EC with 3+2. > > # ceph -s > cluster: > id: 384d7590-d018-11ee-b74c-5b2acfe0b35c > health: HEALTH_WARN > Degraded data redundancy: 2848547/29106793 objects degraded (9.787%), 105 pgs degraded, 132 pgs undersized > > services: > mon: 4 daemons, quorum node1,node5,node4,node2 (age 12h) > mgr: node1.cvknae(active, since 12h), standbys: node4.foomun > mds: 2/2 daemons up, 2 standby > osd: 95 osds: 95 up (since 16h), 95 in (since 21h); 124 remapped pgs > rgw: 2 daemons active (2 hosts, 1 zones) > > data: > volumes: 2/2 healthy > pools: 18 pools, 817 pgs > objects: 6.06M objects, 20 TiB > usage: 30 TiB used, 302 TiB / 332 TiB avail > pgs: 2848547/29106793 objects degraded (9.787%) > 2617833/29106793 objects misplaced (8.994%) > 561 active+clean > 124 active+clean+remapped > 105 active+undersized+degraded > 27 active+undersized > > io: > client: 1.4 MiB/s rd, 4.0 MiB/s wr, 25 op/s rd, 545 op/s wr > > And when using 'ceph config-key ls’ it’s showing old node and osd’s. > > # ceph config-key ls|grep -i 03n > "config-history/135/+osd/host:node3/osd_memory_target", > "config-history/14990/+osd/host:node3/osd_memory_target", > "config-history/14990/-osd/host:node3/osd_memory_target", > "config-history/15003/+osd/host:node3/osd_memory_target", > "config-history/15003/-osd/host:node3/osd_memory_target", > "config-history/15016/+osd/host:node3/osd_memory_target", > "config-history/15016/-osd/host:node3/osd_memory_target", > "config-history/15017/+osd/host:node3/osd_memory_target", > "config-history/15017/-osd/host:node3/osd_memory_target", > "config-history/15022/+osd/host:node3/osd_memory_target", > "config-history/15022/-osd/host:node3/osd_memory_target", > "config-history/15024/+osd/host:node3/osd_memory_target", > "config-history/15024/-osd/host:node3/osd_memory_target", > "config-history/15025/+osd/host:node3/osd_memory_target", > "config-history/15025/-osd/host:node3/osd_memory_target", > "config-history/153/+osd/host:node3/osd_memory_target", > "config-history/153/-osd/host:node3/osd_memory_target", > "config-history/165/+mon.node3/container_image", > "config-history/171/-mon.node3/container_image", > "config-history/176/+client.crash.node3/container_image", > "config-history/182/-client.crash.node3/container_image", > "config-history/4276/+osd/host:node3/osd_memory_target", > "config-history/4276/-osd/host:node3/osd_memory_target", > "config-history/433/+client.ceph-exporter.node3/container_image", > "config-history/439/-client.ceph-exporter.node3/container_image", > "config-history/459/+osd/host:node3/osd_memory_target", > "config-history/459/-osd/host:node3/osd_memory_target", > "config-history/465/+osd/host:node3/osd_memory_target", > "config-history/465/-osd/host:node3/osd_memory_target", > "config-history/4867/+osd/host:node3/osd_memory_target", > "config-history/4867/-osd/host:node3/osd_memory_target", > "config-history/4878/+mon.node3/container_image", > "config-history/4884/-mon.node3/container_image", > "config-history/4889/+client.crash.node3/container_image", > "config-history/4895/-client.crash.node3/container_image", > "config-history/5139/+mds.k8s-dev-cephfs.node3.iebxqn/container_image", > "config-history/5142/-mds.k8s-dev-cephfs.node3.iebxqn/container_image", > "config-history/5150/+client.ceph-exporter.node3/container_image", > "config-history/5156/-client.ceph-exporter.node3/container_image", > "config-history/5179/+osd/host:node3/osd_memory_target", > "config-history/5179/-osd/host:node3/osd_memory_target", > "config-history/5183/+client.rgw.sea-dev.node3.betyqd/rgw_frontends", > "config-history/5189/+osd/host:node3/osd_memory_target", > "config-history/5189/-osd/host:node3/osd_memory_target", > "config-history/6929/-client.rgw.sea-dev.node3.betyqd/rgw_frontends", > "config-history/6933/+osd/host:node3/osd_memory_target", > "config-history/6933/-osd/host:node3/osd_memory_target", > "config-history/9710/+osd/host:node3/osd_memory_target", > "config-history/9710/-osd/host:node3/osd_memory_target", > "config/osd/host:node3/osd_memory_target”, > > > Regards > Dev > >> On Jan 25, 2025, at 4:39 AM, Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx> wrote: >> >> Hi, >> >> I've seen this happening on a test cluster after draining a host that also had a MGR service. Can you check if Eugen's solution here [1] helps in your case ? And maybe investigate 'ceph config-key ls' for any issues in config keys ? >> >> Regards, >> Frédéric. >> >> [1] https://www.spinics.net/lists/ceph-users/msg83667.html <https://www.google.com/url?q=https://www.spinics.net/lists/ceph-users/msg83667.html&source=gmail-imap&ust=1738413580000000&usg=AOvVaw3Zk70LrQ6SrLX02gJ7Cowl> >> >> De : Devender Singh <devender@xxxxxxxxxx> >> Envoyé : samedi 25 janvier 2025 06:27 >> À : Fnu Virender Kumar >> Cc: ceph-users >> Objet : Re: Error ENOENT: Module not found >> >> Thanks for you reply… but those command not working as its an always module..but strange still showing error, >> >> # ceph mgr module enable orchestrator >> module 'orchestrator' is already enabled (always-on) >> >> # ceph orch set backend — returns successfully… >> >> # # ceph orch ls >> Error ENOENT: No orchestrator configured (try `ceph orch set backend`) >> >> Its revolving between same error.. >> >> Root Cause: I removed a hosts and its odd’s and after some time above error started automatically. >> >> Earlier in the had 5 nodes but now 4.. Cluster is showing unclean pg but not doing anything.. >> >> But big error is Error ENOENT: >> >> >> Regards >> Dev >> >> > On Jan 24, 2025, at 4:59 PM, Fnu Virender Kumar <virenderk@xxxxxxxxxxxx> wrote: >> > >> > Did you try >> > >> > Ceph mgr module enable orchestrator >> > Ceph orch set backend >> > Ceph orch ls >> > >> > Check the mgr service daemon as well >> > Ceph -s >> > >> > >> > Regards >> > Virender >> > From: Devender Singh <devender@xxxxxxxxxx> >> > Sent: Friday, January 24, 2025 6:34:43 PM >> > To: ceph-users <ceph-users@xxxxxxx> >> > Subject: Error ENOENT: Module not found >> > >> > >> > Hello all >> > >> > Any quick fix for … >> > >> > root@sea-devnode1:~# ceph orch ls >> > Error ENOENT: Module not found >> > >> > >> > Regards >> > Dev >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |