On 7/24/19 6:02 PM, Sinan Polat wrote: > Hi, > > Why not using backup tools that can do native OpenStack backups? > > We are also using Ceph as the cinder backend on our OpenStack platform. We use CommVault to make our backups. How much data is there in that Ceph cluster? And how does it perform? I assume it exports the RBD images through Cinder and stores them somewhere? But this is probably a complete copy of the RBD image and I just wonder how that scales. A lot of systems are >500TB and it's very difficult to back that up every week or so. Wido > > - Sinan > >> Op 24 jul. 2019 om 17:48 heeft Wido den Hollander <wido@xxxxxxxx> het volgende geschreven: >> >> >> >>> On 7/24/19 4:06 PM, Fabian Niepelt wrote: >>> Hi, thanks for the reply. >>> >>> Am Mittwoch, den 24.07.2019, 15:26 +0200 schrieb Wido den Hollander: >>>> >>>> On 7/24/19 1:37 PM, Fabian Niepelt wrote: >>>>> Hello ceph-users, >>>>> >>>>> I am currently building a Ceph cluster that will serve as a backend for >>>>> Openstack and object storage using RGW. The cluster itself is finished and >>>>> integrated with Openstack and virtual machines for testing are being >>>>> deployed. >>>>> Now I'm a bit stumped on how to effectively backup the Ceph pools. >>>>> My requirements are two weekly backups, of which one must be offline after >>>>> finishing backing up (systems turned powerless). We are expecting about >>>>> 250TB to >>>>> 500TB of data for now. The backups must protect against accidental pool >>>>> deletion/corruption or widespread infection of a cryptovirus. In short: >>>>> Complete >>>>> data loss in the production Ceph cluster. >>>>> >>>>> At the moment, I am facing two issues: >>>>> >>>>> 1. For the cinder pool, I looked into creating snapshots using the ceph CLI >>>>> (so >>>>> they don't turn up in Openstack and cannot be accidentally deleted by users) >>>>> and >>>>> exporting their diffs. But volumes with snapshots created this way cannot be >>>>> removed from Openstack. Does anyone have an idea how to do this better? >>>> >>>> You mean that while you leave the snapshot there OpenStack can't remove it? >>> >>> Yes, that is correct. cinder-volume cannot remove a volume that still has a >>> snapshot. If the snapshot is created by openstack, it will remove the snapshot >>> before removing the volume. But snapshotting directly from ceph will forego >>> Openstack so it will never know about that snapshot's existence. >>> >> >> Ah, yes. That means you would need to remove it manually. >> >>>>> Alternatively, I could do a full export each week, but I am not sure if that >>>>> would be fast enough.. >>>>> >>>> >>>> It probably won't, but the full backup is still the safest way imho. >>>> However: Does this scale? >>>> >>>> You can export multiple RBD images in parallel and store them somewhere >>>> else, but it will still take a long time. >>>> >>>> The export needs to be stored somewhere and then picked up. Or you could >>>> use some magic with Netcat to stream the RBD export to a destination host. >>>> >>> >>> Scaling is also my biggest worry about this. >>> >>>>> 2. My search so far has only turned up backing up RBD pools, but how could I >>>>> backup the pools that are used for object storage? >>>>> >>>> >>>> Not easily. I think you mean RGW? You could try the RGW MultiSite, but >>>> it's difficult. >>>> >>>> A complete DR with Ceph to restore it back to how it was at a given >>>> point in time is a challenge. >>>> >>> >>> Yes, I would like to backup the pools used by the RGW. >> >> Not really an option. You would need to use the RGW MultiSite to >> replicate all data to a second environment. >> >>> >>>>> Of course, I'm also open to completely other ideas on how to backup Ceph and >>>>> would appreciate hearing how you people are doing your backups. >>>> >>>> A lot of time the backups are created inside the VMs on File level. And >>>> there is a second OpenStack+Ceph system which runs a mirror of the VMs >>>> or application. If one burns down it's not the end of the world. >>>> >>>> Trying to backup a Ceph cluster sounds very 'enterprise' and is >>>> difficult to scale as well. >>>> >>> >>> Are those backups saved in Ceph as well? I cannot solely rely on Ceph because we >>> want to protect ourselves against failures in Ceph or a human accidentally or >>> maliciously deletes all pools. >>> >> >> That second OpenStack+Ceph environment is completely different. All the >> VMs are set up twice and using replication and backups on application >> level such things are redundant. >> >> Think about MySQL replication for example. >> >>> From what I'm reading, it seems to be better to maybe implement a backup >>> solution outside of Ceph that our Openstack users can use and not deal with >>> backing up Ceph at all, except its configs to get it running after total >>> desaster... >>> >> >> You could backup OpenStack's MySQL database, the ceph config and then >> backup the data inside the VMs. >> >> It's very difficult to backup data for DR to a certain point of time >> when you go into the >100TB scale. >> >> Wido >> >>>> Wido >>>> >>>>> Any help is much appreciated. >>>>> >>>>> Greetings >>>>> Fabian >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |