Hi Dev, The config-history keys related to node3 are expected to remain even for nodes that you remove, as these keys are not cleaned up upon node deletion. These keys are used by the 'ceph config log' command to list all configuration changes that have occurred over time. The 'ceph status' shows 132 PGs undersized - certainly all related to EC3+2 pools - and 685 other PGs related to replicated pools. With 5 nodes and 22 OSDs per node, that's 132*5/(95+22) = 5 PGs per OSD for EC pools and 685*3/(95+22) = 17 PGs per OSD for replicated pool. With such a low PG per OSD ratio, it is very likely that some pools are not utilizing all OSDs. You might want to increase the number of PGs in these pools to better balance I/O and improve performance. Additionally, ensure that you do not have an even number of MONs. Regards, Frédéric. ----- Le 25 Jan 25, à 20:34, Devender Singh <devender@xxxxxxxxxx> a écrit : > Hello Fredreic > Thanks for your reply, Yes I also faced this issue after draining and removing > of the node. > So used the same command and remove “original_weight” using ceph config-key get > mgr/cephadm/osd_remove_queue and injected file again. Which resolved the orch > issue. > “ Error ENOENT: Module not found - ceph orch commands stoppd working > ceph config-key get mgr/cephadm/osd_remove_queue > osd_remove_queue.json > Then only remove the "original_weight" key from that json and upload it back to > the config-key store: > ceph config-key set mgr/cephadm/osd_remove_queue -i > osd_remove_queue_modified.json > Then fail the mgr: > ceph mgr fail ” > But now issue is, my cluster showing objects misplaced, whereas I had 5 nodes > with host failure domain with R3 pool (size 3 and min 2), EC with 3+2. > # ceph -s > cluster: > id: 384d7590-d018-11ee-b74c-5b2acfe0b35c > health: HEALTH_WARN > Degraded data redundancy: 2848547/29106793 objects degraded (9.787%), 105 pgs > degraded, 132 pgs undersized > services: > mon: 4 daemons, quorum node1,node5,node4,node2 (age 12h) > mgr: node1.cvknae(active, since 12h), standbys: node4.foomun > mds: 2/2 daemons up, 2 standby > osd: 95 osds: 95 up (since 16h), 95 in (since 21h); 124 remapped pgs > rgw: 2 daemons active (2 hosts, 1 zones) > data: > volumes: 2/2 healthy > pools: 18 pools, 817 pgs > objects: 6.06M objects, 20 TiB > usage: 30 TiB used, 302 TiB / 332 TiB avail > pgs: 2848547/29106793 objects degraded (9.787%) > 2617833/29106793 objects misplaced (8.994%) > 561 active+clean > 124 active+clean+remapped > 105 active+undersized+degraded > 27 active+undersized > io: > client: 1.4 MiB/s rd, 4.0 MiB/s wr, 25 op/s rd, 545 op/s wr > And when using 'ceph config-key ls’ it’s showing old node and osd’s. > # ceph config-key ls|grep -i 03n > "config-history/135/+osd/host:node3/osd_memory_target", > "config-history/14990/+osd/host:node3/osd_memory_target", > "config-history/14990/-osd/host:node3/osd_memory_target", > "config-history/15003/+osd/host:node3/osd_memory_target", > "config-history/15003/-osd/host:node3/osd_memory_target", > "config-history/15016/+osd/host:node3/osd_memory_target", > "config-history/15016/-osd/host:node3/osd_memory_target", > "config-history/15017/+osd/host:node3/osd_memory_target", > "config-history/15017/-osd/host:node3/osd_memory_target", > "config-history/15022/+osd/host:node3/osd_memory_target", > "config-history/15022/-osd/host:node3/osd_memory_target", > "config-history/15024/+osd/host:node3/osd_memory_target", > "config-history/15024/-osd/host:node3/osd_memory_target", > "config-history/15025/+osd/host:node3/osd_memory_target", > "config-history/15025/-osd/host:node3/osd_memory_target", > "config-history/153/+osd/host:node3/osd_memory_target", > "config-history/153/-osd/host:node3/osd_memory_target", > "config-history/165/+mon.node3/container_image", > "config-history/171/-mon.node3/container_image", > "config-history/176/+client.crash.node3/container_image", > "config-history/182/-client.crash.node3/container_image", > "config-history/4276/+osd/host:node3/osd_memory_target", > "config-history/4276/-osd/host:node3/osd_memory_target", > "config-history/433/+client.ceph-exporter.node3/container_image", > "config-history/439/-client.ceph-exporter.node3/container_image", > "config-history/459/+osd/host:node3/osd_memory_target", > "config-history/459/-osd/host:node3/osd_memory_target", > "config-history/465/+osd/host:node3/osd_memory_target", > "config-history/465/-osd/host:node3/osd_memory_target", > "config-history/4867/+osd/host:node3/osd_memory_target", > "config-history/4867/-osd/host:node3/osd_memory_target", > "config-history/4878/+mon.node3/container_image", > "config-history/4884/-mon.node3/container_image", > "config-history/4889/+client.crash.node3/container_image", > "config-history/4895/-client.crash.node3/container_image", > "config-history/5139/+mds.k8s-dev-cephfs.node3.iebxqn/container_image", > "config-history/5142/-mds.k8s-dev-cephfs.node3.iebxqn/container_image", > "config-history/5150/+client.ceph-exporter.node3/container_image", > "config-history/5156/-client.ceph-exporter.node3/container_image", > "config-history/5179/+osd/host:node3/osd_memory_target", > "config-history/5179/-osd/host:node3/osd_memory_target", > "config-history/5183/+client.rgw.sea-dev.node3.betyqd/rgw_frontends", > "config-history/5189/+osd/host:node3/osd_memory_target", > "config-history/5189/-osd/host:node3/osd_memory_target", > "config-history/6929/-client.rgw.sea-dev.node3.betyqd/rgw_frontends", > "config-history/6933/+osd/host:node3/osd_memory_target", > "config-history/6933/-osd/host:node3/osd_memory_target", > "config-history/9710/+osd/host:node3/osd_memory_target", > "config-history/9710/-osd/host:node3/osd_memory_target", > "config/osd/host:node3/osd_memory_target”, > Regards > Dev >> On Jan 25, 2025, at 4:39 AM, Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx> >> wrote: >> Hi, >> I've seen this happening on a test cluster after draining a host that also had a >> MGR service. Can you check if Eugen's solution here [1] helps in your case ? >> And maybe investigate 'ceph config-key ls' for any issues in config keys ? >> Regards, >> Frédéric. >> [1] [ >> https://www.google.com/url?q=https://www.spinics.net/lists/ceph-users/msg83667.html&source=gmail-imap&ust=1738413580000000&usg=AOvVaw3Zk70LrQ6SrLX02gJ7Cowl >> | https://www.spinics.net/lists/ceph-users/msg83667.html ] >> De : Devender Singh <devender@xxxxxxxxxx> >> Envoyé : samedi 25 janvier 2025 06:27 >> À : Fnu Virender Kumar >> Cc: ceph-users >> Objet : Re: Error ENOENT: Module not found >> Thanks for you reply… but those command not working as its an always module..but >> strange still showing error, >> # ceph mgr module enable orchestrator >> module 'orchestrator' is already enabled (always-on) >> # ceph orch set backend — returns successfully… >> # # ceph orch ls >> Error ENOENT: No orchestrator configured (try `ceph orch set backend`) >> Its revolving between same error.. >> Root Cause: I removed a hosts and its odd’s and after some time above error >> started automatically. >> Earlier in the had 5 nodes but now 4.. Cluster is showing unclean pg but not >> doing anything.. >> But big error is Error ENOENT: >> Regards >> Dev >> > On Jan 24, 2025, at 4:59 PM, Fnu Virender Kumar <virenderk@xxxxxxxxxxxx> wrote: >> > Did you try >> > Ceph mgr module enable orchestrator >> > Ceph orch set backend >> > Ceph orch ls >> > Check the mgr service daemon as well >> > Ceph -s >> > Regards >> > Virender >> > From: Devender Singh <devender@xxxxxxxxxx> >> > Sent: Friday, January 24, 2025 6:34:43 PM >> > To: ceph-users <ceph-users@xxxxxxx> >> > Subject: Error ENOENT: Module not found >> > Hello all >> > Any quick fix for … >> > root@sea-devnode1:~# ceph orch ls >> > Error ENOENT: Module not found >> > Regards >> > Dev >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |