On Wed, Jun 8, 2022 at 12:36 AM Andreas Teuchert <a.teuchert@xxxxxxxxxxxx> wrote: > > > Hello, > > we're currently evaluating cephfs-mirror. > > We have two data centers with one Ceph cluster in each DC. For now, the > Ceph clusters are only used for CephFS. On each cluster we have one FS > that contains a directory for customer data and below that there are > directories for each customer. The customers access their directories > via CephFS. Generally customers only have data in one data center. > > In order to be able to quickly restore the service, we want to mirror > all customer data in one DC to the Ceph cluster in the other DC. Then, > in case one Ceph cluster becomes unavailable, we would do the following > (let's say the clusters are cluster A and cluster B, and cluster A > became unavailable): > > - "Unmount" the broken mount on clients connecting to cluster A. > - Mount the customer directories from cluster B. > - Repair/restore cluster A. > - Break the mirror relation from cluster A to cluster B. > - Create a mirror relation from cluster B to cluster A (for the data > that should be on A). > - Ensure that cluster A is regularly updated with the current data from > cluster B. > - In a maintenance window: Unmount the directory on all clients that > should connect to cluster A, sync all data (that should be on A) from B > to A, break the mirror relation from B to A, have all clients mount the > directories from A, create a mirror relation from A to B. > > > Setting up the mirroring looks like this: > > # Cluster A: There are two FSs: fs-a for customer data, fs-b-backup for > data mirrored from cluster B > # Cluster B: There are two FSs: fs-b for customer data, fs-a-backup for > data mirrored from cluster A > > (cluster-a)# ls /mnt/fs-a/customers > customer1 customer2 customer3 > > (cluster-b)# ls /mnt/fs-b/customers > customer4 customer5 customer6 > > # Enable mirroring on both clusters > (cluster-a)# ceph orch apply cephfs-mirror > (cluster-a)# ceph mgr module enable mirroring > (cluster-b)# ceph orch apply cephfs-mirror > (cluster-b)# ceph mgr module enable mirroring > > # Setup mirroring of fs-a from cluster A to cluster B > (cluster-b)# ceph fs authorize fs-a-backup client.mirror_a / rwps > (cluster-b)# ceph fs snapshot mirror peer_bootstrap create fs-a-backup > client.mirror_a cluster-a > > (cluster-a)# ceph fs snapshot mirror enable fs-a > (cluster-a)# ceph fs snapshot mirror peer_bootstrap import fs-a <token> > (cluster-a)# ceph fs snapshot mirror add fs-a /customers/customer1 > (cluster-a)# ceph fs snapshot mirror add fs-a /customers/customer2 > (cluster-a)# ceph fs snapshot mirror add fs-a /customers/customer3 > > # Setup mirroring of fs-b from cluster B to cluster A > (cluster-a)# ceph fs authorize fs-b-backup client.mirror_b / rwps > (cluster-a)# ceph fs snapshot mirror peer_bootstrap create fs-b-backup > client.mirror_b cluster-b > > (cluster-b)# ceph fs snapshot mirror enable fs-b > (cluster-b)# ceph fs snapshot mirror peer_bootstrap import fs-b <token> > (cluster-b)# ceph fs snapshot mirror add fs-b /customers/customer4 > (cluster-b)# ceph fs snapshot mirror add fs-b /customers/customer5 > (cluster-b)# ceph fs snapshot mirror add fs-b /customers/customer6 > > # Snapshots for the customer directories are created daily via schedule. > > # Result: Customer data from A mirrored to B and vice-versa > (cluster-a)# ls /mnt/fs-b-backup/customers > customer4 customer5 customer6 > > (cluster-b)# ls /mnt/fs-a-backup/customers > customer1 customer2 customer3 > > > In order to fail over from cluster A to cluster B, we do the following: > > # Create clients for access to fs-a-backup on cluster B > # Mount customer directories from fs-a-backup on cluster B > > # When cluster A is available again > (cluster-a)# ceph fs snapshot mirror remove fs-a /customers/customer1 > (cluster-a)# ceph fs snapshot mirror remove fs-a /customers/customer2 > (cluster-a)# ceph fs snapshot mirror remove fs-a /customers/customer3 > (cluster-a)# ceph fs snapshot mirror disable fs-a > > (cluster-a)# ceph fs authorize fs-a client.mirror_b_failover / rwps > (cluster-a)# ceph fs snapshot mirror peer_bootstrap create fs-a > client.mirror_b_failover cluster-b > (cluster-a)# rmdir /mnt/fs-a/customers/*/.snap/* > > (cluster-b)# setfattr -x ceph.mirror.info /mnt/fs-a-backup > (cluster-b)# ceph fs snapshot mirror enable fs-a-backup > (cluster-b)# ceph fs snapshot mirror peer_bootstrap import fs-a-backup > <token> > (cluster-b)# ceph fs snapshot mirror add fs-a-backup /customers/customer1 > (cluster-b)# ceph fs snapshot mirror add fs-a-backup /customers/customer2 > (cluster-b)# ceph fs snapshot mirror add fs-a-backup /customers/customer3 > > > Now, all clients connect to cluster B and cluster A is only used as a > mirror destination. > At some point we will have a maintenance window and switch all clients > that should be on cluster A back to cluster A. > > > After testing this setup, there are a couple of issues that we have with > the current cephfs-mirror implementation: > > - The xattr ceph.mirror.info seems not to be removed when mirroring is > disabled on a FS (haven't yet gotten around to debug why - looks like a > simple bug). As shown above this can be workarounded by manually > removing the xattr. > - For us it would be convenient if a FS could be mirror source and > destination at the same time. Then we wouldn't need to have two FSs. In > that case for each mirrored directory the destination directory name > would also need to be specified (e.g. mirroring > fs-a:/customers/customer1 to fs-b:/backup/customer1). > - Because now the mirroring between two FSs is always in one direction, > it is not possible to mirror different directories in different > directions. That would be really useful for a failback, because > organizing one big maintenance window where all customers are failed > back at once is very difficult. > - The ceph.dir.layout.pool_namespace xattr is not mirrored - we use this > attribute to separate the RADOS objects of different customers. > Workaround is to manually create the customer directories on the mirror > destination and set the xattr before mirroring starts. > - If files are deleted on the mirror destination, they are not > re-created and there is no notification that there are now files missing. > - Mirroring back to the source cluster after a failover requires that > all snapshots are deleted on the (old) source FS and then everything is > mirrored even though most of the data is already there. The same happens > when failing back. For us this is serveral tens of TBs and serveral > houndred millions of files so having the initial mirroring use the files > that are already present would be a huge improvement. > > > We did all testing with 16.2.7, but as far as I can tell our > observations still apply to 16.2.9 and Quincy. > > Are there any plans for changes that would solve some of those issues > listed above? Unfortunately not at this time. Cephfs-mirror was developed for Red Hat plans which have since changed, and we're not sure when it will come up again — the team wants to make some more comprehensive underlying changes to CephFS that make this scenario work better in general (faster rollback to snapshot points, etc) before we improve our custom mirroring. I've added Venky (who wrote most of the mirror daemon) to check if there's anything you've missed in your points, or small tweaks which did make it in. > > I would be happy to create tickets in Tracker in case that is helpful. Tickets would be good for future reference, even if we won't be taking any actions on them in the near future. :) -Greg _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |