Hi Venky, Thank's a lot for these explanations. I had some trouble when upgrading to v16.2.5. I'm using debian 10 with cephadm and the 16.2.5 containers use generated a lot of network dropped packets (I don't know why) on all my OSD hosts. I encountered also some hangs while reading files in cephfs (may be related with the dropped packets), so I downgraded our 2 clusters to 16.2.4. I will try to resolve my problems with 16.2.5 to benefit from the next enhancements of snapshots mirroring... Kind regards, Arnaud ----- Mail original ----- De: "Venky Shankar" <vshankar@xxxxxxxxxx> À: "arnaud martel" <arnaud.martel@xxxxxxxxxxxxxxxxxxxx> Cc: "ceph-users" <ceph-users@xxxxxxx> Envoyé: Lundi 23 Août 2021 14:39:15 Objet: Re: cephfs snapshots mirroring On Mon, Aug 23, 2021 at 5:36 PM Arnaud MARTEL <arnaud.martel@xxxxxxxxxxxxxxxxxxxx> wrote: > > Hi all, > > I'm not sure to really understand how cephfs snapshots mirroring is supposing to work. > > I have 2 ceph clusters (pacific 16.2.4) and snapshots mirroring is set up for only one directory, /ec42/test, in our cephfs filesytem (it's for test purposes but we plan to use it with about 50-60 directories and 1.5 PB). > I have also set up a pool with erasure coding and configured the layout for my /ec42 directory to use the EC pool (on both clusters). > > I used the following steps to test the snapshot mirroring: > - copy about 70GB in /ec42/test on source cluster > - create a snapshot (mkdir /ec42/test/.snap/snap1) > - remove 5 text files from /ec42/test (the total files size is about 5-10 KB) > - create another snapshot ( mkdir /ec42/test/.snap/snap2) > > What I see during cephfs-mirror execution: > - after snap1 creation, 70 GB are transferred to to target cluster, then snapshot (snap1) is created on target cluster > - after snap2 creation, the remote directory (on target cluster) is emptied, then 70GB are transferred again and, finally, the second snapshot (snap2) is created > > I thought that only the diff between the snapshots would be transferred (or remove) but it seems that all data in the source snapshot are pushed each time. Is it the design of the snapshot mirroring feature or have I missed something? The lack of an efficient snapshot diff feature in CephFS requires the mirror daemon to walk the entire snapshot. That said, with 16.2.5 release, cephfs-mirror only transfers those files that have been modified between two snapshots (16.2.4 does a remote data purge, followed by a full copy), however, to figure out what files have changed it still needs to walk the entire directory tree. Efficient snapshot diff for CephFS is been worked on with priority. > I wanted to use snapshot mirroring to backup our cephfs filesystem but it will be impossible if we have to transfert 1,5PB every day. > And any other suggestion to backup cephfs with 1,5PB would be very helpful... > > Arnaud > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Cheers, Venky _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |