I have been hammering on my setup non stop with only layering, exclusive-lock and deep-flatten enabled. Still can't reproduce the issue. I think this is the ticket. From: "adamb" <adamb@xxxxxxxxxx> To: "dillaman" <dillaman@xxxxxxxxxx> Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> Sent: Thursday, January 28, 2021 3:37:15 PM Subject: Re: RBD-Mirror Snapshot Backup Image Uses Went through the process like 4-5 times now and its looking good. I am going to continue beating on it to make sure. I will report back tomorrow. Nice catch! From: "Jason Dillaman" <jdillama@xxxxxxxxxx> To: "adamb" <adamb@xxxxxxxxxx> Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> Sent: Thursday, January 28, 2021 12:53:50 PM Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses On Thu, Jan 28, 2021 at 10:31 AM Jason Dillaman <jdillama@xxxxxxxxxx> wrote: > > On Wed, Jan 27, 2021 at 7:27 AM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > Doing some more testing. > > > > I can demote the rbd image on the primary, promote on the secondary and the image looks great. I can map it, mount it, and it looks just like it should. > > > > However, the rbd snapshots are still unusable on the secondary even when promoted. I went as far as taking a 2nd snapshot on the rbd before demoting/promoting and that 2nd snapshot still won't work either. Both snapshots look and work great on the primary. > > > > If I request a resync from the secondary the rbd snapshots start working just like the primary, but only if I request a resync. > > > > I get the same exact results whether I am using a 15.2.8 or 16.1 client. Any chance I can get you to retry your test w/ the object-map / fast-diff disabled on the primary image (the features will be propagated to the non-primary image on create / next mirror snapshot)? I was able to recreate the issue on when the object-map is enabled and I'll hopefully get a fix for that issue up for review today. I just want to verify that it is the same issue that you are hitting. > Thanks. I've opened a tracker ticket > (https://tracker.ceph.com/issues/49037) for this. > > > > > > > > > ________________________________ > > From: "adamb" <adamb@xxxxxxxxxx> > > To: "dillaman" <dillaman@xxxxxxxxxx> > > Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > Sent: Tuesday, January 26, 2021 1:51:13 PM > > Subject: Re: RBD-Mirror Snapshot Backup Image Uses > > > > Did some testing with clients running 16.1. I setup two different clients, each one dedicated to its perspective cluster. Running Proxmox, I compiled the latest Pacific 16.1 build. > > > > root@Ccspacificclient:/cephbuild/ceph/build/bin# ./ceph -v > > *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** > > ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) > > > > root@Bunkpacificclient:/cephbuild/ceph/build/bin# ./ceph -v > > *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** > > ceph version 16.1.0-8-g5f17c37f78 (5f17c37f78a331b7a4bf793890f9d324c64183e5) pacific (rc) > > > > Unfortunately, I am hitting the same exact issues using a pacific client. > > > > Would this confirm that its something specific in 15.2.8 on the osd/mon nodes? > > > > > > > > > > > > > > From: "Jason Dillaman" <jdillama@xxxxxxxxxx> > > To: "adamb" <adamb@xxxxxxxxxx> > > Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > Sent: Friday, January 22, 2021 3:44:26 PM > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > On Fri, Jan 22, 2021 at 3:29 PM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > > > I will have to do some looking into how that is done on Proxmox, but most definitely. > > > > Thanks, appreciate it. > > > > > ________________________________ > > > From: "Jason Dillaman" <jdillama@xxxxxxxxxx> > > > To: "adamb" <adamb@xxxxxxxxxx> > > > Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > Sent: Friday, January 22, 2021 3:02:23 PM > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > Any chance you can attempt to repeat the process on the latest master > > > or pacific branch clients (no need to upgrade the MONs/OSDs)? > > > > > > On Fri, Jan 22, 2021 at 2:32 PM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > > > > > The steps are pretty straight forward. > > > > > > > > - Create rbd image of 500G on the primary > > > > - Enable rbd-mirror snapshot on the image > > > > - Map the image on the primary > > > > - Format the block device with ext4 > > > > - Mount it and write out 200-300G worth of data (I am using rsync with some local real data we have) > > > > - Unmap the image from the primary > > > > - Create rdb snapshot > > > > - Create rdb mirror snapshot > > > > - Wait for copy process to complete > > > > - Clone the rdb snapshot on secondary > > > > - Map the image on secondary > > > > - Try to mount on secondary > > > > > > > > Just as a reference. All of my nodes are the same. > > > > > > > > root@Bunkcephtest1:~# ceph --version > > > > ceph version 15.2.8 (8b89984e92223ec320fb4c70589c39f384c86985) octopus (stable) > > > > > > > > root@Bunkcephtest1:~# dpkg -l | grep rbd-mirror > > > > ii rbd-mirror 15.2.8-pve2 amd64 Ceph daemon for mirroring RBD images > > > > > > > > This is pretty straight forward, I don't know what I could be missing here. > > > > > > > > > > > > ________________________________ > > > > From: "Jason Dillaman" <jdillama@xxxxxxxxxx> > > > > To: "adamb" <adamb@xxxxxxxxxx> > > > > Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > Sent: Friday, January 22, 2021 2:11:36 PM > > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > Any chance you could write a small reproducer test script? I can't > > > > repeat what you are seeing and we do have test cases that really > > > > hammer random IO on primary images, create snapshots, rinse-and-repeat > > > > and they haven't turned up anything yet. > > > > > > > > Thanks! > > > > > > > > On Fri, Jan 22, 2021 at 1:50 PM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > > > > > > > I have been doing a lot of testing. > > > > > > > > > > The size of the RBD image doesn't have any effect. > > > > > > > > > > I run into the issue once I actually write data to the rbd. The more data I write out, the larger the chance of reproducing the issue. > > > > > > > > > > I seem to hit the issue of missing the filesystem all together the most, but I have also had a few instances where some of the data was simply missing. > > > > > > > > > > I monitor the mirror status on the remote cluster until the snapshot is 100% copied and also make sure all the IO is done. My setup has no issue maxing out my 10G interconnect during replication, so its pretty obvious once its done. > > > > > > > > > > The only way I have found to resolve the issue is to call a mirror resync on the secondary array. > > > > > > > > > > I can then map the rbd on the primary, write more data to it, snap it again, and I am back in the same position. > > > > > > > > > > ________________________________ > > > > > From: "adamb" <adamb@xxxxxxxxxx> > > > > > To: "dillaman" <dillaman@xxxxxxxxxx> > > > > > Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > > Sent: Thursday, January 21, 2021 3:11:31 PM > > > > > Subject: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > > > Sure thing. > > > > > > > > > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-1 > > > > > SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE > > > > > 12192 TestSnapper1 2 TiB Thu Jan 21 14:15:02 2021 user > > > > > 12595 .mirror.non_primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.34c4a53e-9525-446c-8de6-409ea93c5edd 2 TiB Thu Jan 21 15:05:02 2021 mirror (non-primary peer_uuids:[] 6c26557e-d011-47b1-8c99-34cf6e0c7f2f:12801 copied) > > > > > > > > > > > > > > > root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 > > > > > vm-100-disk-1: > > > > > global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 > > > > > state: up+replaying > > > > > description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611259501,"remote_snapshot_timestamp":1611259501,"replay_state":"idle"} > > > > > service: admin on Bunkcephmon1 > > > > > last_update: 2021-01-21 15:06:24 > > > > > peer_sites: > > > > > name: ccs > > > > > state: up+stopped > > > > > description: local image is primary > > > > > last_update: 2021-01-21 15:06:23 > > > > > > > > > > > > > > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE > > > > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE > > > > > /dev/nbd0 > > > > > root@Bunkcephtest1:~# blkid /dev/nbd0 > > > > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > > > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. > > > > > > > > > > > > > > > Primary still looks good. > > > > > > > > > > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper1 CephTestPool1/vm-100-disk-1-CLONE > > > > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE > > > > > /dev/nbd0 > > > > > root@Ccscephtest1:~# blkid /dev/nbd0 > > > > > /dev/nbd0: UUID="830b8e05-d5c1-481d-896d-14e21d17017d" TYPE="ext4" > > > > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2 > > > > > root@Ccscephtest1:~# cat /proc/mounts | grep nbd0 > > > > > /dev/nbd0 /usr2 ext4 rw,relatime 0 0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: "Jason Dillaman" <jdillama@xxxxxxxxxx> > > > > > To: "adamb" <adamb@xxxxxxxxxx> > > > > > Cc: "Eugen Block" <eblock@xxxxxx>, "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > > Sent: Thursday, January 21, 2021 3:01:46 PM > > > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > > > On Thu, Jan 21, 2021 at 11:51 AM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > > > > > > > > > I was able to trigger the issue again. > > > > > > > > > > > > - On the primary I created a snap called TestSnapper for disk vm-100-disk-1 > > > > > > - Allowed the next RBD-Mirror scheduled snap to complete > > > > > > - At this point the snapshot is showing up on the remote side. > > > > > > > > > > > > root@Bunkcephtest1:~# rbd mirror image status CephTestPool1/vm-100-disk-1 > > > > > > vm-100-disk-1: > > > > > > global_id: a04e92df-3d64-4dc4-8ac8-eaba17b45403 > > > > > > state: up+replaying > > > > > > description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"local_snapshot_timestamp":1611247200,"remote_snapshot_timestamp":1611247200,"replay_state":"idle"} > > > > > > service: admin on Bunkcephmon1 > > > > > > last_update: 2021-01-21 11:46:24 > > > > > > peer_sites: > > > > > > name: ccs > > > > > > state: up+stopped > > > > > > description: local image is primary > > > > > > last_update: 2021-01-21 11:46:28 > > > > > > > > > > > > root@Ccscephtest1:/etc/pve/priv# rbd snap ls --all CephTestPool1/vm-100-disk-1 > > > > > > SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE > > > > > > 11532 TestSnapper 2 TiB Thu Jan 21 11:21:25 2021 user > > > > > > 11573 .mirror.primary.a04e92df-3d64-4dc4-8ac8-eaba17b45403.9525e4eb-41c0-499c-8879-0c7d9576e253 2 TiB Thu Jan 21 11:35:00 2021 mirror (primary peer_uuids:[debf975b-ebb8-432c-a94a-d3b101e0f770]) > > > > > > > > > > > > Seems like the sync is complete, So I then clone it, map it and attempt to mount it. > > > > > > > > > > Can you run "snap ls --all" on the non-primary cluster? The > > > > > non-primary snapshot will list its status. On my cluster (with a much > > > > > smaller image): > > > > > > > > > > # > > > > > # CLUSTER 1 > > > > > # > > > > > $ rbd --cluster cluster1 create --size 1G mirror/image1 > > > > > $ rbd --cluster cluster1 mirror image enable mirror/image1 snapshot > > > > > Mirroring enabled > > > > > $ rbd --cluster cluster1 device map -t nbd mirror/image1 > > > > > /dev/nbd0 > > > > > $ mkfs.ext4 /dev/nbd0 > > > > > mke2fs 1.45.5 (07-Jan-2020) > > > > > Discarding device blocks: done > > > > > Creating filesystem with 262144 4k blocks and 65536 inodes > > > > > Filesystem UUID: 50e0da12-1f99-4d45-b6e6-5f7a7decaeff > > > > > Superblock backups stored on blocks: > > > > > 32768, 98304, 163840, 229376 > > > > > > > > > > Allocating group tables: done > > > > > Writing inode tables: done > > > > > Creating journal (8192 blocks): done > > > > > Writing superblocks and filesystem accounting information: done > > > > > $ blkid /dev/nbd0 > > > > > /dev/nbd0: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" > > > > > BLOCK_SIZE="4096" TYPE="ext4" > > > > > $ rbd --cluster cluster1 snap create mirror/image1@fs > > > > > Creating snap: 100% complete...done. > > > > > $ rbd --cluster cluster1 mirror image snapshot mirror/image1 > > > > > Snapshot ID: 6 > > > > > $ rbd --cluster cluster1 snap ls --all mirror/image1 > > > > > SNAPID NAME > > > > > SIZE PROTECTED TIMESTAMP > > > > > NAMESPACE > > > > > 5 fs > > > > > 1 GiB Thu Jan 21 14:50:24 2021 > > > > > user > > > > > 6 .mirror.primary.f9f692b8-2405-416c-9247-5628e303947a.39722e17-f7e6-4050-acf0-3842a5620d81 > > > > > 1 GiB Thu Jan 21 14:50:51 2021 mirror (primary > > > > > peer_uuids:[cd643f30-4982-4caf-874d-cf21f6f4b66f]) > > > > > > > > > > # > > > > > # CLUSTER 2 > > > > > # > > > > > > > > > > $ rbd --cluster cluster2 mirror image status mirror/image1 > > > > > image1: > > > > > global_id: f9f692b8-2405-416c-9247-5628e303947a > > > > > state: up+replaying > > > > > description: replaying, > > > > > {"bytes_per_second":1140872.53,"bytes_per_snapshot":17113088.0,"local_snapshot_timestamp":1611258651,"remote_snapshot_timestamp":1611258651,"replay_state":"idle"} > > > > > service: mirror.0 on cube-1 > > > > > last_update: 2021-01-21 14:51:18 > > > > > peer_sites: > > > > > name: cluster1 > > > > > state: up+stopped > > > > > description: local image is primary > > > > > last_update: 2021-01-21 14:51:27 > > > > > $ rbd --cluster cluster2 snap ls --all mirror/image1 > > > > > SNAPID NAME > > > > > SIZE PROTECTED TIMESTAMP > > > > > NAMESPACE > > > > > 5 fs > > > > > 1 GiB Thu Jan 21 14:50:52 > > > > > 2021 user > > > > > 6 .mirror.non_primary.f9f692b8-2405-416c-9247-5628e303947a.0a13b822-0508-47d6-a460-a8cc4e012686 > > > > > 1 GiB Thu Jan 21 14:50:53 2021 mirror (non-primary > > > > > peer_uuids:[] 9824df2b-86c4-4264-a47e-cf968efd09e1:6 copied) > > > > > $ rbd --cluster cluster2 --rbd-default-clone-format 2 clone > > > > > mirror/image1@fs mirror/image2 > > > > > $ rbd --cluster cluster2 device map -t nbd mirror/image2 > > > > > /dev/nbd1 > > > > > $ blkid /dev/nbd1 > > > > > /dev/nbd1: UUID="50e0da12-1f99-4d45-b6e6-5f7a7decaeff" > > > > > BLOCK_SIZE="4096" TYPE="ext4" > > > > > $ mount /dev/nbd1 /mnt/ > > > > > $ mount | grep nbd > > > > > /dev/nbd1 on /mnt type ext4 (rw,relatime,seclabel) > > > > > > > > > > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE > > > > > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > > > > > /dev/nbd0 > > > > > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > > > > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. > > > > > > > > > > > > On the primary still no issues > > > > > > > > > > > > root@Ccscephtest1:/etc/pve/priv# rbd clone CephTestPool1/vm-100-disk-1@TestSnapper CephTestPool1/vm-100-disk-1-CLONE > > > > > > root@Ccscephtest1:/etc/pve/priv# rbd-nbd map CephTestPool1/vm-100-disk-1-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > > > > > /dev/nbd0 > > > > > > root@Ccscephtest1:/etc/pve/priv# mount /dev/nbd0 /usr2 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > From: "Jason Dillaman" <jdillama@xxxxxxxxxx> > > > > > > To: "adamb" <adamb@xxxxxxxxxx> > > > > > > Cc: "Eugen Block" <eblock@xxxxxx>, "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > > > Sent: Thursday, January 21, 2021 9:42:26 AM > > > > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > > > > > On Thu, Jan 21, 2021 at 9:40 AM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > After the resync finished. I can mount it now. > > > > > > > > > > > > > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > > > > > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > > > > > > /dev/nbd0 > > > > > > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > > > > > > > > > > > > > Makes me a bit nervous how it got into that position and everything appeared ok. > > > > > > > > > > > > We unfortunately need to create the snapshots that are being synced as > > > > > > a first step, but perhaps there are some extra guardrails we can put > > > > > > on the system to prevent premature usage if the sync status doesn't > > > > > > indicate that it's complete. > > > > > > > > > > > > > ________________________________ > > > > > > > From: "Jason Dillaman" <jdillama@xxxxxxxxxx> > > > > > > > To: "adamb" <adamb@xxxxxxxxxx> > > > > > > > Cc: "Eugen Block" <eblock@xxxxxx>, "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > > > > Sent: Thursday, January 21, 2021 9:25:11 AM > > > > > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > > > > > > > On Thu, Jan 21, 2021 at 8:34 AM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > > > When cloning the snapshot on the remote cluster I can't see my ext4 filesystem. > > > > > > > > > > > > > > > > Using the same exact snapshot on both sides. Shouldn't this be consistent? > > > > > > > > > > > > > > Yes. Has the replication process completed ("rbd mirror image status > > > > > > > CephTestPool1/vm-100-disk-0")? > > > > > > > > > > > > > > > Primary Site > > > > > > > > root@Ccscephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > > > > > > > 10621 TestSnapper1 2 TiB Thu Jan 21 08:15:22 2021 user > > > > > > > > > > > > > > > > root@Ccscephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > > > > > > > root@Ccscephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > > > > > > > /dev/nbd0 > > > > > > > > root@Ccscephtest1:~# mount /dev/nbd0 /usr2 > > > > > > > > > > > > > > > > Secondary Site > > > > > > > > root@Bunkcephtest1:~# rbd snap ls --all CephTestPool1/vm-100-disk-0 | grep TestSnapper1 > > > > > > > > 10430 TestSnapper1 2 TiB Thu Jan 21 08:20:08 2021 user > > > > > > > > > > > > > > > > root@Bunkcephtest1:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool1/vm-100-disk-0-CLONE > > > > > > > > root@Bunkcephtest1:~# rbd-nbd map CephTestPool1/vm-100-disk-0-CLONE --id admin --keyring /etc/ceph/ceph.client.admin.keyring > > > > > > > > /dev/nbd0 > > > > > > > > root@Bunkcephtest1:~# mount /dev/nbd0 /usr2 > > > > > > > > mount: /usr2: wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > From: "adamb" <adamb@xxxxxxxxxx> > > > > > > > > To: "dillaman" <dillaman@xxxxxxxxxx> > > > > > > > > Cc: "Eugen Block" <eblock@xxxxxx>, "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > > > > > Sent: Wednesday, January 20, 2021 3:42:46 PM > > > > > > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > > > > > > > > > Awesome information. I new I had to be missing something. > > > > > > > > > > > > > > > > All of my clients will be far newer than mimic so I don't think that will be an issue. > > > > > > > > > > > > > > > > Added the following to my ceph.conf on both clusters. > > > > > > > > > > > > > > > > rbd_default_clone_format = 2 > > > > > > > > > > > > > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > > > > > > > root@Bunkcephmon2:~# rbd ls CephTestPool2 > > > > > > > > vm-100-disk-0-CLONE > > > > > > > > > > > > > > > > I am sure I will be back with more questions. Hoping to replace our Nimble storage with Ceph and NVMe. > > > > > > > > > > > > > > > > Appreciate it! > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > From: "Jason Dillaman" <jdillama@xxxxxxxxxx> > > > > > > > > To: "adamb" <adamb@xxxxxxxxxx> > > > > > > > > Cc: "Eugen Block" <eblock@xxxxxx>, "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > > > > > Sent: Wednesday, January 20, 2021 3:28:39 PM > > > > > > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > > > > > > > > > On Wed, Jan 20, 2021 at 3:10 PM Adam Boyhan <adamb@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > That's what I though as well, specially based on this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Note > > > > > > > > > > > > > > > > > > You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writeable clones in another pool. > > > > > > > > > > > > > > > > > > root@Bunkcephmon2:~# rbd clone CephTestPool1/vm-100-disk-0@TestSnapper1 CephTestPool2/vm-100-disk-0-CLONE > > > > > > > > > 2021-01-20T15:06:35.854-0500 7fb889ffb700 -1 librbd::image::CloneRequest: 0x55c7cf8417f0 validate_parent: parent snapshot must be protected > > > > > > > > > > > > > > > > > > root@Bunkcephmon2:~# rbd snap protect CephTestPool1/vm-100-disk-0@TestSnapper1 > > > > > > > > > rbd: protecting snap failed: (30) Read-only file system > > > > > > > > > > > > > > > > You have two options: (1) protect the snapshot on the primary image so > > > > > > > > that the protection status replicates or (2) utilize RBD clone v2 > > > > > > > > which doesn't require protection but does require Mimic or later > > > > > > > > clients [1]. > > > > > > > > > > > > > > > > > > > > > > > > > > From: "Eugen Block" <eblock@xxxxxx> > > > > > > > > > To: "adamb" <adamb@xxxxxxxxxx> > > > > > > > > > Cc: "ceph-users" <ceph-users@xxxxxxx>, "Matt Wilder" <matt.wilder@xxxxxxxxxx> > > > > > > > > > Sent: Wednesday, January 20, 2021 3:00:54 PM > > > > > > > > > Subject: Re: Re: RBD-Mirror Snapshot Backup Image Uses > > > > > > > > > > > > > > > > > > But you should be able to clone the mirrored snapshot on the remote > > > > > > > > > cluster even though it’s not protected, IIRC. > > > > > > > > > > > > > > > > > > > > > > > > > > > Zitat von Adam Boyhan <adamb@xxxxxxxxxx>: > > > > > > > > > > > > > > > > > > > Two separate 4 node clusters with 10 OSD's in each node. Micron 9300 > > > > > > > > > > NVMe's are the OSD drives. Heavily based on the Micron/Supermicro > > > > > > > > > > white papers. > > > > > > > > > > > > > > > > > > > > When I attempt to protect the snapshot on a remote image, it errors > > > > > > > > > > with read only. > > > > > > > > > > > > > > > > > > > > root@Bunkcephmon2:~# rbd snap protect > > > > > > > > > > CephTestPool1/vm-100-disk-0@TestSnapper1 > > > > > > > > > > rbd: protecting snap failed: (30) Read-only file system > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > > > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > > > > _______________________________________________ > > > > > > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > > > > > > > > > > > [1] https://ceph.io/community/new-mimic-simplified-rbd-image-cloning/ > > > > > > > > > > > > > > > > -- > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Jason > > > > > > > > > > > > > > > > > > > > -- > > > > > Jason > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > > > > > > > > > > > -- > > > > Jason > > > > > > > > > > > > -- > > > Jason > > > > > > > > -- > > Jason > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > -- > Jason -- Jason _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |