This can't be done in a very nice way currently. There's actually an open PR against main to allow setting the crush location for mons in the service spec specifically because others found that this was annoying as well. What I think should work as a workaround is, go to the host where the mon that failed to join the quorum but failed due to lack of crush location is, open the /var/lib/ceph/<fsid>/mon.<mon-id>/unit.run file, then at the very end of the last line (the last line should be a long podman/docker run command) append "--set-crush-location <crush-loc>". Then, still on that host, a "systemctl restart <mon-service>" where mon-service is the systemd unit listed for the mon in "cephadm ls --no-detail". That should allow the monitor to at least join the quorum as it now has a crush location, and then you should be able to make other alterations a bit easier. On Fri, Dec 2, 2022 at 7:40 AM Sake Paulusma <sake1989@xxxxxxxxxxx> wrote: > I succesfully setup a stretched cluster, except the CRUSH rule mentioned > in the docs wasn't correct. The parameters for "min_size" and "max_size" > should be removed, or else the rule can't be imported. > Second there should be a mention about setting the monitor crush location > takes sometime and know other ceph command can be used. > > But now I need to replace a few monitors (it's virtualized and machines > need to be replaced). I use cephadm and have a label "mon" which is > assigned to the monitor services. > With the command "ceph orch host add <hostname> <ip-address> --labels=mon" > I add normally a new monitor to the cluster. > > Only this results in the following error in de logs: > 12/2/22 1:27:19 PM [INF] <hostname> attempted to join from <hostname> > [v2:<ip-address>:3300/0,v1:<ip-address>:6789/0]; but lacks a crush_location > for stretch mode > > Next I tried to set the CRUSH location like used in the docs for stretch > mode with the command "ceph mon set_location <hostname> datacenter=<dc1>". > This only results in the following error: > Error ENOENT: mon.oqsoel11437 does not exist > > So how can I add/replace a monitor in a stretched cluster? > > Best regards, > Sake > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |