On 7/24/19 4:06 PM, Fabian Niepelt wrote: > Hi, thanks for the reply. > > Am Mittwoch, den 24.07.2019, 15:26 +0200 schrieb Wido den Hollander: >> >> On 7/24/19 1:37 PM, Fabian Niepelt wrote: >>> Hello ceph-users, >>> >>> I am currently building a Ceph cluster that will serve as a backend for >>> Openstack and object storage using RGW. The cluster itself is finished and >>> integrated with Openstack and virtual machines for testing are being >>> deployed. >>> Now I'm a bit stumped on how to effectively backup the Ceph pools. >>> My requirements are two weekly backups, of which one must be offline after >>> finishing backing up (systems turned powerless). We are expecting about >>> 250TB to >>> 500TB of data for now. The backups must protect against accidental pool >>> deletion/corruption or widespread infection of a cryptovirus. In short: >>> Complete >>> data loss in the production Ceph cluster. >>> >>> At the moment, I am facing two issues: >>> >>> 1. For the cinder pool, I looked into creating snapshots using the ceph CLI >>> (so >>> they don't turn up in Openstack and cannot be accidentally deleted by users) >>> and >>> exporting their diffs. But volumes with snapshots created this way cannot be >>> removed from Openstack. Does anyone have an idea how to do this better? >> >> You mean that while you leave the snapshot there OpenStack can't remove it? > > Yes, that is correct. cinder-volume cannot remove a volume that still has a > snapshot. If the snapshot is created by openstack, it will remove the snapshot > before removing the volume. But snapshotting directly from ceph will forego > Openstack so it will never know about that snapshot's existence. > Ah, yes. That means you would need to remove it manually. >>> Alternatively, I could do a full export each week, but I am not sure if that >>> would be fast enough.. >>> >> >> It probably won't, but the full backup is still the safest way imho. >> However: Does this scale? >> >> You can export multiple RBD images in parallel and store them somewhere >> else, but it will still take a long time. >> >> The export needs to be stored somewhere and then picked up. Or you could >> use some magic with Netcat to stream the RBD export to a destination host. >> > > Scaling is also my biggest worry about this. > >>> 2. My search so far has only turned up backing up RBD pools, but how could I >>> backup the pools that are used for object storage? >>> >> >> Not easily. I think you mean RGW? You could try the RGW MultiSite, but >> it's difficult. >> >> A complete DR with Ceph to restore it back to how it was at a given >> point in time is a challenge. >> > > Yes, I would like to backup the pools used by the RGW. Not really an option. You would need to use the RGW MultiSite to replicate all data to a second environment. > >>> Of course, I'm also open to completely other ideas on how to backup Ceph and >>> would appreciate hearing how you people are doing your backups. >> >> A lot of time the backups are created inside the VMs on File level. And >> there is a second OpenStack+Ceph system which runs a mirror of the VMs >> or application. If one burns down it's not the end of the world. >> >> Trying to backup a Ceph cluster sounds very 'enterprise' and is >> difficult to scale as well. >> > > Are those backups saved in Ceph as well? I cannot solely rely on Ceph because we > want to protect ourselves against failures in Ceph or a human accidentally or > maliciously deletes all pools. > That second OpenStack+Ceph environment is completely different. All the VMs are set up twice and using replication and backups on application level such things are redundant. Think about MySQL replication for example. > From what I'm reading, it seems to be better to maybe implement a backup > solution outside of Ceph that our Openstack users can use and not deal with > backing up Ceph at all, except its configs to get it running after total > desaster... > You could backup OpenStack's MySQL database, the ceph config and then backup the data inside the VMs. It's very difficult to backup data for DR to a certain point of time when you go into the >100TB scale. Wido >> Wido >> >>> Any help is much appreciated. >>> >>> Greetings >>> Fabian >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |