On Wed, Sep 13, 2017 at 10:54 AM, Wido den Hollander <wido@xxxxxxxx> wrote: > >> Op 13 september 2017 om 10:38 schreef Dan van der Ster <dan@xxxxxxxxxxxxxx>: >> >> >> Hi Blair, >> >> You can add/remove mons on the fly -- connected clients will learn >> about all of the mons as the monmap changes and there won't be any >> downtime as long as the quorum is maintained. >> >> There is one catch when it comes to OpenStack, however. >> Unfortunately, OpenStack persists the mon IP addresses at volume >> creation time. So next time you hard reboot a VM, it will try >> connecting to the old set of mons. >> Whatever you have in ceph.conf on the hypervisors is irrelevant (after >> a volume was created) -- libvirt uses the IPs in each instance's xml >> directly. >> > > That's why I always recommend people to use DNS and preferably a Round Robin DNS record to overcome these situations. > > That should work with OpenStack as well. > > ceph-mon.storage.local. AAAA 2001:db8::101 > ceph-mon.storage.local. AAAA 2001:db8::102 > ceph-mon.storage.local. AAAA 2001:db8::103 > > And then use *ceph-mon.storage.local* in your OpenStack configuration a MON. > Does that work? Last time I checked, it works like this when a new volume is attached: - OpenStack connects to ceph using ceph.conf, DNS, whatever... - Retrieve the monmap. - Extract the list of IPs from the monmap. - Persist the IPs in the block-device-mapping table. I still find that logic here: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L163 Next time you hard-reboot the VM, it will connect to the IPs directly -- not use ceph.conf or DNS. -- Dan > Wido > >> There is an old ticket here: https://bugs.launchpad.net/cinder/+bug/1452641 >> It's recently gone unassigned, but there is a new proposed fix here: >> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118040.html >> >> As of today, you will need to manually update nova's >> block-device-mapping table for every volume when you re-ip the mons. >> >> Cheers, Dan >> >> >> On Wed, Sep 13, 2017 at 4:57 AM, Blair Bethwaite >> <blair.bethwaite@xxxxxxxxx> wrote: >> > Hi all, >> > >> > We're looking at readdressing the mons (moving to a different subnet) >> > on one of our clusters. Most of the existing clients are OpenStack >> > guests on Libvirt+KVM and we have a major upgrade to do for those in >> > coming weeks that will mean they have to go down briefly, that will >> > give us an opportunity to update their libvirt config to point them at >> > new mon addresses. We plan to do the upgrade in a rolling fashion and >> > thus need to keep Ceph services up the whole time. >> > >> > So question is, can we for example have our existing 3 mons on network >> > N1, add another 2 mons on network N2, reconfigure VMs to use the 2 new >> > mon addresses, all whilst not impacting running clients. You can >> > assume we'll setup routing such that the new mons can talk to the old >> > mons, OSDs, and vice-versa. >> > >> > Perhaps flipping the question on its head - if you configure a librbd >> > client with only a subset of mon addresses will it *only* talk to >> > those mons, or will it just use that config to bootstrap and then talk >> > to any mons that are up in the current map? Or likewise, is there >> > anything the client has to talk to the mon master for? >> > >> > -- >> > Cheers, >> > ~Blairo >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@xxxxxxxxxxxxxx >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |