Hi, I had to change the IPs of my cluster some time ago. The process was quite easy. I don't understand what you mean with configuring and deleting static routes. The easies way is if the router allows (at least for the change) all traffic between the old and the new network. I did the following steps. 1. Add the new IP Network space separated to the "public network" line in your ceph.conf 2. OSDS: stop you OSDs on the first node. Reconfigure the host network and start your OSDs again. Repeat this for all hosts one by one 3. MON: stop and remove one mon from cluster, delete all data in /var/ceph/mon/mon.<id> reconfigure the host network. Create the new mon instance (don't forget the "mon host" entrys in your ceph.conf and your clients as well) Of course this requires at least 3 Mons in your cluster! After 2 of 5 Mons in my cluster I added the new mon adresses to my clients and restarted them. 4. MGR: stop the mgr daemon. reconfigure the host network. Start the mgr daemon one by one I wouldn't recomend the "messy way" to reconfigure your mons. removing and adding mons to the cluster is quite easy and in my opinion the most secure. The complet IP change in our cluster worked without outage while the cluster was in production. I hope I could help you. Regards Manuel On Fri, 19 Jul 2019 10:22:37 +0000 "ST Wong (ITSC)" <ST@xxxxxxxxxxxxxxxx> wrote: > Hi all, > > Our cluster has to change to new IP range in same VLAN: 10.0.7.0/24 > -> 10.0.18.0/23, while IP address on private network for OSDs > remains unchanged. I wonder if we can do that in either one following > ways: > > ===================== > > 1. > > a. Define static route for 10.0.18.0/23 on each node > > b. Do it one by one: > > For each monitor/mgr: > > - remove from cluster > > - change IP address > > - add static route to original IP range 10.0.7.0/24 > > - delete static route for 10.0.18.0/23 > > - add back to cluster > > For each OSD: > > - stop OSD daemons > > - change IP address > > - add static route to original IP range 10.0.7.0/24 > > - delete static route for 10.0.18.0/23 > > - start OSD daemons > > c. Clean up all static routes defined. > > > > 2. > > a. Export and update monmap using the messy way as described in > http://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-mons/ > > > > ceph mon getmap -o {tmp}/{filename} > > monmaptool -rm node1 -rm node2 ... --rm node n {tmp}/{filename} > > monmaptool -add node1 v2:10.0.18.1:3330,v1:10.0.18.1:6789 -add node2 > v2:10.0.18.2:3330,v1:10.0.18.2:6789 ... --add nodeN > v2:10.0.18.N:3330,v1:10.0.18.N:6789 {tmp}/{filename} > > > > b. stop entire cluster daemons and change IP addresses > > > c. For each mon node: ceph-mon -I {mon-id} -inject-monmap > {tmp}/{filename} > > > > d. Restart cluster daemons. > > > > 3. Or any better method... > ===================== > > Would anyone please help? Thanks a lot. > Rgds > /st wong > -- Manuel Lausch Systemadministrator Storage Services 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 | 76135 Karlsruhe | Germany Phone: +49 721 91374-1847 E-Mail: manuel.lausch@xxxxxxxx | Web: www.1und1.de Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452 Geschäftsführer: Alexander Charles, Thomas Ludwig, Jan Oetjen, Sascha Vollmer Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |