Hi, noted and thanks a lot.
Best Rgds
/stwong
-----Original Message-----
From: Ricardo Dias <rdias@xxxxxxxx>
Sent: Thursday, July 25, 2019 8:47 PM
To: ST Wong (ITSC) <ST@xxxxxxxxxxxxxxxx>; Manuel Lausch <manuel.lausch@xxxxxxxx>; ceph-users@xxxxxxxxxxxxxx
Subject: Re: Please help: change IP address of a cluster
Hi,
The monmaptool has a different option to add an address vector. To add an address vector do this:
monmaptool --addv node1 [v2:10.0.1.1:3330,v1:10.0.1.1:6789] {tmp}/{filename}
The --addv option does not appear in the usage description but the fix is already under way: https://github.com/ceph/ceph/pull/29307
Cheers,
--
Ricardo Dias
Senior Software Engineer - Storage Team
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
On 24/07/19 07:20, ST Wong (ITSC) wrote:
> Hi,
>
> Thanks for your help.
> I changed IP addresses of OSD nodes successfully. When changing IP address on MON, it works except that the MON only listens on v2 port 3300 after adding the MON back to the cluster. Previously the MON listens on both v1 (6789) and v2 (3300).
> Besides, can't add both v1 and v2 entries manually using monmaptool like following. Only the 2nd one will get added.
>
> monmaptool -add node1 v1:10.0.1.1:6789, v2:10.0.1.1:3330
> {tmp}/{filename}
>
> the monmap now looks like following:
>
> min_mon_release 14 (nautilus)
> 0: [v2:10.0.1.92:3300/0,v1:10.0.1.92:6789/0] mon.cmon2
> 1: [v2:10.0.1.93:3300/0,v1:10.0.1.93:6789/0] mon.cmon2
> 2: [v2:10.0.1.94:3300/0,v1:10.0.1.94:6789/0] mon.cmon3
> 3: [v2:10.0.1.95:3300/0,v1:10.0.1.95:6789/0] mon.cmon4
> 4: v2:10.0.1.97:3300/0 mon.cmon5 <--- the MON being removed/added
>
> Although it's okay to use v2 only, I'm afraid I've missed some steps and messed the cluster up. Any advice?
> Thanks again.
>
> Best Regards,
> /stwong
>
> -----Original Message-----
> From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> On Behalf Of
> Manuel Lausch
> Sent: Tuesday, July 23, 2019 7:32 PM
> To: ceph-users@xxxxxxxxxxxxxx
> Subject: Re: Please help: change IP address of a cluster
>
> Hi,
>
> I had to change the IPs of my cluster some time ago. The process was quite easy.
>
> I don't understand what you mean with configuring and deleting static
> routes. The easies way is if the router allows (at least for the
> change) all traffic between the old and the new network.
>
> I did the following steps.
>
> 1. Add the new IP Network space separated to the "public network" line
> in your ceph.conf
>
> 2. OSDS: stop you OSDs on the first node. Reconfigure the host network
> and start your OSDs again. Repeat this for all hosts one by one
>
> 3. MON: stop and remove one mon from cluster, delete all data in /var/ceph/mon/mon.<id> reconfigure the host network. Create the new mon instance (don't forget the "mon host" entrys in your ceph.conf and your clients as well) Of course this requires at least 3 Mons in your cluster!
> After 2 of 5 Mons in my cluster I added the new mon adresses to my clients and restarted them.
>
> 4. MGR: stop the mgr daemon. reconfigure the host network. Start the
> mgr daemon one by one
>
>
> I wouldn't recomend the "messy way" to reconfigure your mons. removing and adding mons to the cluster is quite easy and in my opinion the most secure.
>
> The complet IP change in our cluster worked without outage while the cluster was in production.
>
> I hope I could help you.
>
> Regards
> Manuel
>
>
>
> On Fri, 19 Jul 2019 10:22:37 +0000
> "ST Wong (ITSC)" <ST@xxxxxxxxxxxxxxxx> wrote:
>
>> Hi all,
>>
>> Our cluster has to change to new IP range in same VLAN: 10.0.7.0/24
>> -> 10.0.18.0/23, while IP address on private network for OSDs
>> remains unchanged. I wonder if we can do that in either one following
>> ways:
>>
>> =====================
>>
>> 1.
>>
>> a. Define static route for 10.0.18.0/23 on each node
>>
>> b. Do it one by one:
>>
>> For each monitor/mgr:
>>
>> - remove from cluster
>>
>> - change IP address
>>
>> - add static route to original IP range 10.0.7.0/24
>>
>> - delete static route for 10.0.18.0/23
>>
>> - add back to cluster
>>
>> For each OSD:
>>
>> - stop OSD daemons
>>
>> - change IP address
>>
>> - add static route to original IP range 10.0.7.0/24
>>
>> - delete static route for 10.0.18.0/23
>>
>> - start OSD daemons
>>
>> c. Clean up all static routes defined.
>>
>>
>>
>> 2.
>>
>> a. Export and update monmap using the messy way as described in
>> http://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-mons/
>>
>>
>>
>> ceph mon getmap -o {tmp}/{filename}
>>
>> monmaptool -rm node1 -rm node2 ... --rm node n {tmp}/{filename}
>>
>> monmaptool -add node1 v2:10.0.18.1:3330,v1:10.0.18.1:6789 -add node2
>> v2:10.0.18.2:3330,v1:10.0.18.2:6789 ... --add nodeN
>> v2:10.0.18.N:3330,v1:10.0.18.N:6789 {tmp}/{filename}
>>
>>
>>
>> b. stop entire cluster daemons and change IP addresses
>>
>>
>> c. For each mon node: ceph-mon -I {mon-id} -inject-monmap
>> {tmp}/{filename}
>>
>>
>>
>> d. Restart cluster daemons.
>>
>>
>>
>> 3. Or any better method...
>> =====================
>>
>> Would anyone please help? Thanks a lot.
>> Rgds
>> /st wong
>>
>
>
>
> --
> Manuel Lausch
>
> Systemadministrator
> Storage Services
>
> 1&1 Mail & Media Development & Technology GmbH | Brauerstraße 48 |
> 76135 Karlsruhe | Germany Phone: +49 721 91374-1847
> E-Mail: manuel.lausch@xxxxxxxx | Web: www.1und1.de
>
> Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 5452
>
> Geschäftsführer: Alexander Charles, Thomas Ludwig, Jan Oetjen, Sascha
> Vollmer
>
>
> Member of United Internet
>
> Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden.
>
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail.
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
