Confirming Tyler's description, we had to do lots of database
manipulation in order to get the new IPs into the connection parameters.
Since you already added the new monitors, there's not much else you can
do. But I would have suggested to rather reinstall the MONs instead of
adding new ones as Tyler already stated.
Am 29.11.24 um 13:19 schrieb Tyler Stachecki:
On Fri, Nov 29, 2024, 5:33 AM Adrien Georget <adrien.georget@xxxxxxxxxxx>
wrote:
Hello,
We are using Ceph as a storage backend for Openstack (Cinder, Nova,
Glance, Manila) and we are replacing old hardware hosting Ceph monitors
(MON,MGR,MDS) to new ones.
I have already added the new ones in production, monitors successfully
joined the quorum and new MGR/MDS are standby.
For the monitors, I'm sure that the monmap is already up to date and
Openstack clients are already aware of the change and it should not be a
problem when I will next shut down the old monitors.
The ceph.conf will be updated in all Openstack controllers to replace
"mon host" with the new ones before shutting old mons down.
But I have some doubts with the resilience of Openstack Manila service
because the IP addresses of the monitors look hardcoded in the export
location of the manila share :
The manila show command returns for example :
| export_locations | |
| | path =
134.158.208.140:6789,134.158.208.141:6789,134.158.208.142:6789:/volumes/EC_manila/_nogroup/7a6c05d9-2fea-43b1-a6d4-06eec1e384f2
|
| | share_instance_id =
7a6c05d9-2fea-43b1-a6d4-06eec1e384f2 |
Has anyone already done this kind of migration in the past and can
confirm my doubts?
Is there any process to update shares?
Cheers,
Adrien
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
I can't speak for Manila, but for Cinder/Glance/Nova this is a bit of a
headache. Unfortunately, the mon IPs get hard coded there as well, both in
the database and in the libvirt XML. Go to any nova-compute node with a
Ceph-backed Cinder volume (or Nova image, including config drives) attached
to it and run `virsh dumpxml <UUID>` and you'll see it.
Unfortunately, changing all of the mon IPs will result in the situation
where you can neither live-migrate your VMs nor will you be able to
start/hard reboot VMs until volumes are detached and attached with the new
monitor IPs.
The only way we found around this with zero downtime was to rebuild _some_
of the ceph-mons with new IPs, and then leverage some custom patches (which
I can share) that rewrite the libvirt and database info during a
live-migration (so, in essence, we had to live-migrate each VM once in
order to pull this off) with the new set of intended mon IPs (not the ones
currently in ceph.conf).
If you don't require live-migration or don't use it, you can probably get
away with just doing some database updates (carefully!). The VMs do observe
monmap changes at runtime like any other RADOS client - it's only when you
try to perform control plane actions against them that it becomes a
problem, because it'll use the mon IPs in the database (which are old) and
not from ceph.conf in that case.
Thanks,
Tyler
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
