Hello Adam et al., Thank you for the reply and suggestions. I will work on deploying the additional services using .yaml and the instructions you suggested. First I need to get rid of these two stuck daemons. The ‘ ceph orch rm’ command fails to remove the service. If I use ‘ceph orch daemon rm’ the service gets removed but reappears shortly later on another host. Also adding –force doesn’t change the outcome. Initial state, notice the one daemon is running on cephmon03 & the other on osd26: [ceph: root@osd16 /]# ceph orch ps | grep error mds.cephmon03.local osd16.local osd17.local osd18.local.cephmon03.talhqb cephmon03.local error 3m ago 36m <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mds.cephmon03.local osd16.local osd17.local osd18.local.osd26.drymjr osd26.local error 3m ago 46m <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> Trying Adam’s suggestion: [ceph: root@osd16 /]# ceph orch rm "mds.cephmon03.local osd16.local osd17.local osd18.local.onl26.drymjr" Failed to remove service. <mds.cephmon03.local osd16.local osd17.local osd18.local.osd26.drymjr> was not found. Service gets removed: [ceph: root@osd16 /]# ceph orch daemon rm "mds.cephmon03.local osd16.local osd17.local osd18.local.osd26.drymjr" Removed mds.cephmon03.local osd16.local osd17.local osd18.local.osd26.drymjr from host 'osd26.local' Ceph will stay in this state with only 1 failed daemon for a while, if I remove the 2nd one they both come back: [ceph: root@osd16 /]# ceph orch ps | grep error mds.cephmon03.local osd16.local osd17.local osd18.local.cephmon03.talhqb cephmon03.local error 4m ago 37m <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> Removing the 2nd daemon: [ceph: root@osd16 /]# ceph orch daemon rm "mds.cephmon03.local osd16.local osd17.local osd18.local.cephmon03.talhqb" Removed mds.cephmon03.local osd16.local osd17.local osd18.local.cephmon03.talhqb from host 'cephmon03.local' After a few minutes….notice one daemon on cephmon03 & the other osd30. This seems random [ceph: root@osd16 /]# ceph orch ps | grep error mds.cephmon03.local osd16.local osd17.local osd18.local.cephmon03.pwtvcw cephmon03.local error 53s ago 12m <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mds.cephmon03.local osd16.local osd17.local osd18.local.osd30.ythzbh osd30.local error 40s ago 15m <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> Any further suggestions are helpful. Thanks, -Michael From: Adam King <adking@xxxxxxxxxx> Sent: Wednesday, January 19, 2022 3:37 PM To: Poat, Michael <mpoat@xxxxxxx> Cc: ceph-users <ceph-users@xxxxxxx> Subject: Re: Help Removing Failed Cephadm Daemon(s) - MDS Deployment Issue Hello Michael, If you're trying to remove all the mds daemons in this mds "cephmon03.local osd16.local osd17.local osd18.local" I think the command would be "ceph orch rm "mds.cephmon03.local osd16.local osd17.local osd18.local"" (note the quotes around that mds.cepmon . . . since cephadm thinks this is the service id rather than the placement as I think you intended. I'd check "ceph orch apply mds -h" to see the order it takes the args). Also, as for getting the mds or any other daemon where you want them, I'd recommend taking a look at https://docs.ceph.com/en/pacific/cephadm/services/#service-specification<https://urldefense.com/v3/__https:/docs.ceph.com/en/pacific/cephadm/services/*service-specification__;Iw!!P4SdNyxKAPE!R-fa0E_N0R4J566OABLofDU2As0p7eZBcVi0G3WFI_KcrXSZKDnUlDRU95sXkg$> and the subsequent https://docs.ceph.com/en/pacific/cephadm/services/#daemon-placement<https://urldefense.com/v3/__https:/docs.ceph.com/en/pacific/cephadm/services/*daemon-placement__;Iw!!P4SdNyxKAPE!R-fa0E_N0R4J566OABLofDU2As0p7eZBcVi0G3WFI_KcrXSZKDnUlDQdX098Pw$> (it's pacific docs rather than octopus but most of it should still apply). Basically, cephadm works in a declarative fashion so if you tell it to apply mds to host1 then tell it to apply mds to host2 it will ONLY put the mds on host2 in the end as the second command overwrites the first. If you want the mds on both host1 and host2 you need to tell it that in the placement up front. The easiest way to do so is to create a yaml spec and apply it as outlined in the linked documentation (it can be done with cli directly as well but it's trickier to format the placement correctly so I recommend yaml). - Adam On Wed, Jan 19, 2022 at 2:58 PM Poat, Michael <mpoat@xxxxxxx<mailto:mpoat@xxxxxxx>> wrote: Hello, In a production cluster am running Ceph Octopus on a CentOS 7 based cluster. Use only for CephFS. I used Cephadm for deployment w/ docker-ce. I have 3 (intended) MDS/MON/MGR nodes + 30 storage nodes 4 OSD each (120 OSD total). I have a few issues with Ceph Octopus that have led me to where I am now. I will try to be specific and avoid taking you far down the rabbit hole. When I deployed the cluster, the 3 intended MDS/MON/MGR (named: cephmon01, cephmon02, cephmon03) were not available so 3 of the OSD nodes took place to run these services (names: osd16, osd17, osd18). osd{16..18} are running 4xOSD/1xMDS/1xMON/1xMGR while the remaining osd{01..30} are running 4xOSD only. In efforts to migrate the MDS/MON/MGR services off the 3 OSD nodes, specifically the MDS services, to the 'cephmon{01..03}' nodes I have tried deploying the MDS service onto the 3 additional nodes using the "ceph orch apply mds *<hostname>*" command but nothing happens. I believe this was due to the limit of 3 MDS's set similar to https://docs.ceph.com/en/octopus/cephadm/install/#deploy-additional-monitors-optional<https://urldefense.com/v3/__https:/docs.ceph.com/en/octopus/cephadm/install/*deploy-additional-monitors-optional__;Iw!!P4SdNyxKAPE!R-fa0E_N0R4J566OABLofDU2As0p7eZBcVi0G3WFI_KcrXSZKDnUlDTQAc7Elw$>. In a test cluster I found that if simply increase the number of MDS servers using 'ceph orch apply mds 6, it will deploy 3 additional MDS services on 3 random OSD nodes and not on my cephmon{01..03} nodes, hence the reason I tried to deploy the service on a specific node. I didn't try to increase the number of MDS this way on my production cluster. Additionally, I found that the host labels in 'ceph orch host ls' don't do anything and going that route by adding labels to hosts is a façade. THE ISSUE: What I tried was a command like this (which worked with deployment of additional MGR servers) ceph orch apply mds "cephmon03.local osd16.local osd17.local osd18.local" the idea being to deploy the MDS service on 1 new machine and 3 existing nodes...however this failed miserably and I am now stuck with the following [ceph: root@osd16 /]# ceph -s cluster: id: 1111111111111111111111111111111111 health: HEALTH_WARN 2 failed cephadm daemon(s) services: mon: 5 daemons, quorum osd16.local osd17,osd18,osd19,osd20 (age 15h) mgr: cephmon03.xjyrgm(active, since 5w), standbys: osd18.voydwg, osdl17.tqyxxn, osd16.nkzhik mds: cephfs:1 {0=cephfs.osd16.dlircu=up:active} 2 up:standby osd: 120 osds: 120 up (since 2h), 120 in (since 2w) [ceph: root@osd16 /]# ceph orch ls mds NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID mds.cephfs 3/3 3m ago 5M count:3 docker.io/ceph/ceph:v15<https://urldefense.com/v3/__http:/docker.io/ceph/ceph:v15__;!!P4SdNyxKAPE!R-fa0E_N0R4J566OABLofDU2As0p7eZBcVi0G3WFI_KcrXSZKDnUlDSJ3yPqtA$> 2cf504fded39 mds.cephmon03.local osd16.local osd17.local osd18.local 0/2 3m ago 5w count:2 docker.io/ceph/ceph:v15<https://urldefense.com/v3/__http:/docker.io/ceph/ceph:v15__;!!P4SdNyxKAPE!R-fa0E_N0R4J566OABLofDU2As0p7eZBcVi0G3WFI_KcrXSZKDnUlDSJ3yPqtA$> <unknown> [ceph: root@osd16 /]# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID ... mds.cephmon03.local osd16.local osd17.local osd18.local.osd14.pdgelh osd14.local error 4m ago 6d <unknown> docker.io/ceph/ceph:v15<https://urldefense.com/v3/__http:/docker.io/ceph/ceph:v15__;!!P4SdNyxKAPE!R-fa0E_N0R4J566OABLofDU2As0p7eZBcVi0G3WFI_KcrXSZKDnUlDSJ3yPqtA$> <unknown> <unknown> mds.cephmon03.local osd16.local osd17.local osd18.local.osd26.zdlarv osd26.local error 3m ago 6d <unknown> docker.io/ceph/ceph:v15<https://urldefense.com/v3/__http:/docker.io/ceph/ceph:v15__;!!P4SdNyxKAPE!R-fa0E_N0R4J566OABLofDU2As0p7eZBcVi0G3WFI_KcrXSZKDnUlDSJ3yPqtA$> <unknown> <unknown> ... If I try to do something like ceph orch daemon rm 'mds.cephmon03.local osd16.local osd17.local osd18.local.osd14.pdgelh' (with or without a -force). The daemon will just reappear on a different HOST when checking back with ceph orch ps For starters, any idea on how to remove the failed MDS deployment command? It looks like instead of adding additional nodes to the existing mds.cephfs, it tried creating a 2nd parallel MDS deployment. Thanks, -Michael _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |