Hi Blair, You can add/remove mons on the fly -- connected clients will learn about all of the mons as the monmap changes and there won't be any downtime as long as the quorum is maintained. There is one catch when it comes to OpenStack, however. Unfortunately, OpenStack persists the mon IP addresses at volume creation time. So next time you hard reboot a VM, it will try connecting to the old set of mons. Whatever you have in ceph.conf on the hypervisors is irrelevant (after a volume was created) -- libvirt uses the IPs in each instance's xml directly. There is an old ticket here: https://bugs.launchpad.net/cinder/+bug/1452641 It's recently gone unassigned, but there is a new proposed fix here: http://lists.openstack.org/pipermail/openstack-dev/2017-June/118040.html As of today, you will need to manually update nova's block-device-mapping table for every volume when you re-ip the mons. Cheers, Dan On Wed, Sep 13, 2017 at 4:57 AM, Blair Bethwaite <blair.bethwaite@xxxxxxxxx> wrote: > Hi all, > > We're looking at readdressing the mons (moving to a different subnet) > on one of our clusters. Most of the existing clients are OpenStack > guests on Libvirt+KVM and we have a major upgrade to do for those in > coming weeks that will mean they have to go down briefly, that will > give us an opportunity to update their libvirt config to point them at > new mon addresses. We plan to do the upgrade in a rolling fashion and > thus need to keep Ceph services up the whole time. > > So question is, can we for example have our existing 3 mons on network > N1, add another 2 mons on network N2, reconfigure VMs to use the 2 new > mon addresses, all whilst not impacting running clients. You can > assume we'll setup routing such that the new mons can talk to the old > mons, OSDs, and vice-versa. > > Perhaps flipping the question on its head - if you configure a librbd > client with only a subset of mon addresses will it *only* talk to > those mons, or will it just use that config to bootstrap and then talk > to any mons that are up in the current map? Or likewise, is there > anything the client has to talk to the mon master for? > > -- > Cheers, > ~Blairo > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |