Dear Jason, On 2019-09-10 18:50, Jason Dillaman wrote: > On Tue, Sep 10, 2019 at 12:25 PM Oliver Freyermuth > <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: >> >> Dear Cephalopodians, >> >> I have two questions about RBD mirroring. >> >> 1) I can not get it to work - my setup is: >> >> - One cluster holding the live RBD volumes and snapshots, in pool "rbd", cluster name "ceph", >> running latest Mimic. >> I ran "rbd mirror pool enable rbd pool" on that cluster and created a cephx user "rbd_mirror" with (is there a better way?): >> ceph auth get-or-create client.rbd_mirror mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow pool rbd r' -o ceph.client.rbd_mirror.keyring --cluster ceph >> In that pool, two images have the journaling feature activated, all others have it disabled still (so I would expect these two to be mirrored). > > You can just use "mon 'profile rbd' osd 'profile rbd'" for the caps -- > but you definitely need more than read-only permissions to the remote > cluster since it needs to be able to create snapshots of remote images > and update/trim the image journals. these profiles really make life a lot easier. I should have thought of them rather than "guessing" a potentially good configuration... > >> - Another (empty) cluster running latest Nautilus, cluster name "ceph", pool "rbd". >> I've used the dashboard to activate mirroring for the RBD pool, and then added a peer with cluster name "ceph-virt", cephx-ID "rbd_mirror", filled in the mons and key created above. >> I've then run: >> ceph auth get-or-create client.rbd_mirror_backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow pool rbd rwx' -o client.rbd_mirror_backup.keyring --cluster ceph >> and deployed that key on the rbd-mirror machine, and started the service with: > > Please use "mon 'profile rbd-mirror' osd 'profile rbd'" for your caps [1]. That did the trick (in combination with the above)! Again a case of PEBKAC: I should have read the documentation until the end, clearly my fault. It works well now, even though it seems to run a bit slow (~35 MB/s for the initial sync when everything is 1 GBit/s), but that may also be caused by combination of some very limited hardware on the receiving end (which will be scaled up in the future). A single host with 6 disks, replica 3 and a RAID controller which can only do RAID0 and not JBOD is certainly not ideal, so commit latency may cause this slow bandwidth. > >> systemctl start ceph-rbd-mirror@rbd_mirror_backup.service >> >> After this, everything looks fine: >> # rbd mirror pool info >> Mode: pool >> Peers: >> UUID NAME CLIENT >> XXXXXXXXXXX ceph-virt client.rbd_mirror >> >> The service also seems to start fine, but logs show (debug rbd_mirror=20): >> >> rbd::mirror::ClusterWatcher:0x5575e2a7d390 resolve_peer_config_keys: retrieving config-key: pool_id=2, pool_name=rbd, peer_uuid=XXXXXXXXXXX >> rbd::mirror::Mirror: 0x5575e29c7240 update_pool_replayers: enter >> rbd::mirror::Mirror: 0x5575e29c7240 update_pool_replayers: restarting failed pool replayer for uuid: XXXXXXXXXXX cluster: ceph-virt client: client.rbd_mirror >> rbd::mirror::PoolReplayer: 0x5575e2a7da20 init: replaying for uuid: XXXXXXXXXXX cluster: ceph-virt client: client.rbd_mirror >> rbd::mirror::PoolReplayer: 0x5575e2a7da20 init_rados: error connecting to remote peer uuid: XXXXXXXXXXX cluster: ceph-virt client: client.rbd_mirror: (95) Operation not supported >> rbd::mirror::ServiceDaemon: 0x5575e29c8d70 add_or_update_callout: pool_id=2, callout_id=2, callout_level=error, text=unable to connect to remote cluster > > If it's still broken after fixing your caps above, perhaps increase > debugging for "rados", "monc", "auth", and "ms" to see if you can > determine the source of the op not supported error. > >> I already tried storing the ceph.client.rbd_mirror.keyring (i.e. from the cluster with the live images) on the rbd-mirror machine explicitly (i.e. not only in mon config storage), >> and after doing that: >> rbd -m mon_ip_of_ceph_virt_cluster --id=rbd_mirror ls >> works fine. So it's not a connectivity issue. Maybe a permission issue? Or did I miss something? >> >> Any idea what "operation not supported" means? >> It's unclear to me whether things should work well using Mimic with Nautilus, and enabling pool mirroring but only having journaling on for two images is a supported case. > > Yes and yes. > >> 2) Since there is a performance drawback (about 2x) for journaling, is it also possible to only mirror snapshots, and leave the live volumes alone? >> This would cover the common backup usecase before deferred mirroring is implemented (or is it there already?). > > This is in-development right now and will hopefully land for the > Octopus release. That would be very cool. Just to clarify: You mean the "real" deferred mirroring, not a "snapshot only" mirroring? Is it already clear if this will require Octopous (or a later release) on both ends, or only on the receiving side? Since I got you personally, I have two bonus questions. 1) Your talk: https://events.static.linuxfound.org/sites/events/files/slides/Disaster%20Recovery%20and%20Ceph%20Block%20Storage-%20Introducing%20Multi-Site%20Mirroring.pdf mentions "rbd journal object flush age", which I'd translate with something like the "commit" mount option on a classical file system - correct? I don't find this switch documented anywhere, though - is there experience with it / what's the default? 2) I read I can run more than one rbd-mirror with Mimic/Nautilus. Do they load-balance the images, or "only" failover in case one of them dies? Cheers and many thanks for the quick and perfect help! Oliver > >> Cheers and thanks in advance, >> Oliver >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > [1] https://docs.ceph.com/docs/master/rbd/rbd-mirroring/#rbd-mirror-daemon > > -- > Jason >
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
