Re: journal based mirroring works but snapshot based not

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Changing it to rx-tx seems to do the whole magic.
After restarting rbd-mirror on the backup site the image was synced.

Thanks a lot for your perfect and fast support.
Solving problems this way is fun.

Yves

-----Ursprüngliche Nachricht-----
Von: Jason Dillaman <jdillama@xxxxxxxxxx> 
Gesendet: Freitag, 24. Juli 2020 23:11
An: Yves Kretzschmar-Schwipper <yveskretzschmar@xxxxxx>
Cc: dillaman <dillaman@xxxxxxxxxx>; ceph-users <ceph-users@xxxxxxx>
Betreff: Re:  Re: journal based mirroring works but snapshot based not

On Fri, Jul 24, 2020 at 3:49 PM Yves Kretzschmar-Schwipper <yveskretzschmar@xxxxxx> wrote:
>
> Yes, there are some:
>
> 2020-07-24T21:46:25.974+0200 7f932fccd700 10 rbd::mirror::RemotePollPoller: 0x5628339d92b0 handle_task:
> 2020-07-24T21:46:25.974+0200 7f93304ce700 10 rbd::mirror::RemotePollPoller: 0x5628339d92b0 get_mirror_uuid:
> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 
> rbd::mirror::RemotePollPoller: 0x5628339d92b0 handle_get_mirror_uuid: 
> r=0
> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 
> rbd::mirror::RemotePollPoller: 0x5628339d92b0 handle_get_mirror_uuid: 
> remote_mirror_uuid=ce9e8f12-c8e2-4c16-bd74-f28f29d405a3
> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 rbd::mirror::RemotePollPoller: 0x5628339d92b0 mirror_peer_ping:
> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 
> rbd::mirror::RemotePollPoller: 0x5628339d92b0 handle_mirror_peer_ping: 
> r=0
> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 rbd::mirror::RemotePollPoller: 0x5628339d92b0 mirror_peer_list:
> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 
> rbd::mirror::RemotePollPoller: 0x5628339d92b0 handle_mirror_peer_list: 
> r=0
> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 
> rbd::mirror::RemotePollPoller: 0x5628339d92b0 handle_mirror_peer_list: 
> remote_mirror_peer_uuid=

Yeah, it's not finding itself (by site-name) in the remote peer cluster. Presuming this was a log from your "backup" site's rbd-mirror daemon, I think the issue is that your peer record on "master" for "backup" is listed as a "tx-only" peer, which means that "backup" will only transmit images to "master". Can you try changing the peer direction on "master" to "rx-only" (i.e. it will only receive images from "master") or "rx-tx" for bi-directional mirroring?

> 2020-07-24T21:46:25.978+0200 7f931dca9700 10 rbd::mirror::RemotePollPoller: 0x5628339d92b0 schedule_task:
>
>
> -----Ursprüngliche Nachricht-----
> Von: Jason Dillaman <jdillama@xxxxxxxxxx>
> Gesendet: Freitag, 24. Juli 2020 21:34
> An: Yves Kretzschmar-Schwipper <yveskretzschmar@xxxxxx>
> Cc: ceph-users <ceph-users@xxxxxxx>
> Betreff: Re:  Re: journal based mirroring works but 
> snapshot based not
>
> On Fri, Jul 24, 2020 at 3:22 PM <yveskretzschmar@xxxxxx> wrote:
> >
> > Hi Jason,
> >
> > now I'm a bit confused ...
> >
> > I recreated the peers again with names clear:
> >
> > #main site:
> > rbd mirror pool info testpool
> > Mode: image
> > Site Name: master
> > Peer Sites:
> > UUID: e7443ebd-b9e2-47ab-bcfe-f40cb75a6294
> > Name: backup
> > Mirror UUID: e0d03424-1913-4e0b-90dd-61defabae51a
> > Direction: tx-only
> >
> > #backup site:
> > rbd --cluster backup mirror pool info testpool
> > Mode: image
> > Site Name: backup
> > Peer Sites:
> > UUID: a7ffac0b-09e3-427c-a3a1-5f001a55dcf1
> > Name: master
> > Mirror UUID:
> > Direction: rx-tx
> > Client: client.admin
> >
> >
> > In my opinion this looks ok, but the error seems the same:
> >
> > 2020-07-24T21:14:21.559+0200 7fb9d709a700 -1
> > rbd::mirror::image_replayer::snapshot::Replayer: 0x56122bb43800 init:
> > failed to retrieve mirror peer uuid from remote pool
> > 2020-07-24T21:14:21.559+0200 7fb9e28b1700 -1
> > rbd::mirror::image_replayer::snapshot::Replayer: 0x56122bb43800
> > get_replay_status: replay not running
> > 2020-07-24T21:14:22.515+0200 7fb9e28b1700  5 rbd::mirror::Instances:
> > 0x56122af49d40 handle_acked: instance_ids=[10464751]
> > 2020-07-24T21:14:27.519+0200 7fb9e28b1700  5 rbd::mirror::Instances:
> > 0x56122af49d40 handle_acked: instance_ids=[10464751]
> > 2020-07-24T21:14:31.559+0200 7fb9e28b1700 -1
> > rbd::mirror::image_replayer::snapshot::Replayer: 0x56122bb43800
> > get_replay_status: replay not running
> > 2020-07-24T21:14:32.519+0200 7fb9e28b1700  5 rbd::mirror::Instances:
> > 0x56122af49d40 handle_acked: instance_ids=[10464751]
>
> Turn your rbd-mirror debug level up from 5 to at least 10 and retry.
> Every 30 seconds or so there should be a log message from the "RemotePoolPoller" which is responsible for pulling the mirror peer uuid from the remote cluster.
>
> >
> > If I disable mirroring on the testimage on main site, the mirror daemon tells me:
> >
> > 2020-07-24T21:15:58.775+0200 7f3cd6f3b700 -1
> > rbd::mirror::image_replayer::snapshot::Replayer: 0x55fbad17c800
> > handle_unregister_remote_update_watcher: failed to unregister remote 
> > update watcher: (2) No such file or directory
> > 2020-07-24T21:15:58.775+0200 7f3cd6f3b700 -1
> > rbd::mirror::image_replayer::snapshot::Replayer: 0x55fbad17c800
> > handle_unregister_local_update_watcher: failed to unregister local 
> > update watcher: (2) No such file or directory
> >
> > The created image on the backup site doesn't get removed.
> >
> >
> > Journal based mirroring works without any problem on the same image.
> >
> >
> > Thanks for any hints
> > Yves
> >
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Betreff: Re: Re:  Re: journal based mirroring works but 
> > snapshot based not
> >
> > You need to have unique names for your site -- so "ceph" is not an appropriate name.  Right now, your backup set is named "ceph" and you have a peer named "ceph".
> >
> > On Fri, Jul 24, 2020 at 8:43 AM Yves wrote:
> > >
> > > Runing "rbd mirror pool peer set testpool 0622b647-2c24-4840-93d0-efdee180ab15 site-name backup"
> > > get's me:
> > > 2020-07-24T14:07:57.205+0200 7fc414af53c0 -1 librbd::api::Mirror:
> > > peer_site_set_name: failed to update site
> > > '0622b647-2c24-4840-93d0-efdee180ab15': (17) File exists
> > >
> > > So I deleted peers again and recreated with same site-name.
> > >
> > > On main site it looks this way:
> > > rbd mirror pool info testpool
> > > Mode: image
> > > Site Name: ceph
> > > Peer Sites:
> > > UUID: e68b09de-1d2c-4ec6-9350-a6ccad26e1b7
> > > Name: ceph
> > > Mirror UUID: 4d7f87f4-47be-46dd-85f1-79caa3fa23da
> > > Direction: tx-only
> > >
> > >
> > > On backup site:
> > > rbd --cluster backup mirror pool info testpool
> > > Mode: image
> > > Site Name: ceph
> > > Peer Sites:
> > > UUID: 399e902c-deee-498c-bcdc-1c4f7a59433a
> > > Name: ceph
> > > Mirror UUID:
> > > Direction: rx-tx
> > > Client: client.admin
> > >
> > >
> > > But situation didn't change.
> > >
> > >
> > > Gesendet: Freitag, 24. Juli 2020 um 14:04 Uhr
> > > Von: "Jason Dillaman"
> > > Cc: "ceph-users" <ceph-users@xxxxxxx>
> > > >
> > > > Hi,
> > > >
> > > > this is the main site:
> > > >
> > > > rbd mirror pool info testpool
> > > > Mode: image
> > > > Site Name: ceph
> > > >
> > > > Peer Sites:
> > > >
> > > > UUID: 1f1877cb-5753-4a0e-8b8c-5e5547c0619e
> > > > Name: backup
> > > > Mirror UUID: e9e2c4a0-1900-4db6-b828-e655be5ed9d8
> > > > Direction: tx-only
> > > >
> > > >
> > > > and the backup site:
> > > >
> > > > rbd --cluster backup mirror pool info testpool
> > > > Mode: image
> > > > Site Name: backup
> > > >
> > > > Peer Sites:
> > > >
> > > > UUID: 0622b647-2c24-4840-93d0-efdee180ab15
> > > > Name: ceph
> > > > Direction: rx-only
> > > > Client: client.rbd-mirror-peer
> > >
> > > At the backup site, run "rbd mirror pool peer set testpool
> > > 0622b647-2c24-4840-93d0-efdee180ab15 site-name backup" to update 
> > > the site-name to match.
> > >
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe 
> > > > send an email to ceph-users-leave@xxxxxxx
> > > >
> > >
> > >
> > > --
> > > Jason
> > >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an 
> > email to ceph-users-leave@xxxxxxx
>
>
>
> --
> Jason
>
>


--
Jason

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux