Hi there, We are trying to setup a rbd-mirror replication and after the setup, everything looks good but images are not replicating. Can some please please help? Thanks, -Vikas root@remote:/var/log/ceph# rbd --cluster cephdr mirror pool info nfs Mode: pool Peers: UUID NAME CLIENT bcd54bc5-cd08-435f-a79a-357bce55011d ceph client.mirrorprod root@local:/etc/ceph# rbd mirror pool info nfs Mode: pool Peers: UUID NAME CLIENT 612151cf-f70d-49d0-94e2-a7b850a53e4f cephdr client.mirrordr root@local:/etc/ceph# rbd info nfs/test01 rbd image 'test01': size 102400 kB in 25 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.11cd3c238e1f29 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling flags: journal: 11cd3c238e1f29 mirroring state: enabled mirroring global id: 06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7 mirroring primary: true root@remote:/var/log/ceph# rbd --cluster cephdr mirror pool status nfs --verbose health: OK images: 0 total root@remote:/var/log/ceph# rbd info nfs/test01 rbd: error opening image test01: (2) No such file or directory root@remote:/var/log/ceph# ceph -s --cluster cephdr cluster: id: ade49174-1f84-4c3c-a93c-b293c3655c93 health: HEALTH_WARN noout,nodeep-scrub flag(s) set services: mon: 3 daemons, quorum nidcdvtier1a,nidcdvtier2a,nidcdvtier3a mgr: nidcdvtier1a(active), standbys: nidcdvtier2a osd: 12 osds: 12 up, 12 in flags noout,nodeep-scrub rbd-mirror: 1 daemon active data: pools: 5 pools, 640 pgs objects: 1.32M objects, 5.03TiB usage: 10.1TiB used, 262TiB / 272TiB avail pgs: 640 active+clean io: client: 170B/s rd, 0B/s wr, 0op/s rd, 0op/s wr 2019-04-05 12:07:29.720742 7f0fa5e284c0 0 ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable), process rbd-mirror, pid 3921391 2019-04-05 12:07:29.721752 7f0fa5e284c0 0 pidfile_write: ignore empty --pid-file 2019-04-05 12:07:29.726580 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon: 0x560200d29bb0 ServiceDaemon: 2019-04-05 12:07:29.732654 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon: 0x560200d29bb0 init: 2019-04-05 12:07:29.734920 7f0fa5e284c0 1 mgrc service_daemon_register rbd-mirror.admin metadata {arch=x86_64,ceph_version=ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable),cpu=Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz,distro=ubuntu,distro_description=Ubuntu 14.04.5 LTS,distro_version=14.04,hostname=nidcdvtier3a,instance_id=464360,kernel_description=#93 SMP Sat Jun 17 04:01:23 EDT 2017,kernel_version=3.19.0-85-vtier,mem_swap_kb=67105788,mem_total_kb=131999112,os=Linux} 2019-04-05 12:07:29.735779 7f0fa5e284c0 20 rbd::mirror::Mirror: 0x560200d27f90 run: enter 2019-04-05 12:07:29.735793 7f0fa5e284c0 20 rbd::mirror::ClusterWatcher:0x560200dcd930 refresh_pools: enter 2019-04-05 12:07:29.735809 7f0f77fff700 20 rbd::mirror::ImageDeleter: 0x560200dcd9c0 run: enter 2019-04-05 12:07:29.735819 7f0f77fff700 20 rbd::mirror::ImageDeleter: 0x560200dcd9c0 run: waiting for delete requests 2019-04-05 12:07:29.739019 7f0fa5e284c0 10 rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is disabled for pool docnfs 2019-04-05 12:07:29.741090 7f0fa5e284c0 10 rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is disabled for pool doccifs 2019-04-05 12:07:29.742620 7f0fa5e284c0 10 rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is disabled for pool fcp-dr 2019-04-05 12:07:29.744446 7f0fa5e284c0 10 rbd::mirror::ClusterWatcher:0x560200dcd930 read_pool_peers: mirroring is disabled for pool cifs 2019-04-05 12:07:29.746958 7f0fa5e284c0 20 rbd::mirror::ServiceDaemon: 0x560200d29bb0 add_pool: pool_id=8, pool_name=nfs 2019-04-05 12:07:29.748181 7f0fa5e284c0 20 rbd::mirror::Mirror: 0x560200d27f90 update_pool_replayers: enter 2019-04-05 12:07:29.748212 7f0fa5e284c0 20 rbd::mirror::Mirror: 0x560200d27f90 update_pool_replayers: starting pool replayer for uuid: bcd54bc5-cd08-435f-a79a-357bce55011d cluster: ceph client: client.mirrorprod 2019-04-05 12:07:29.748249 7f0fa5e284c0 20 rbd::mirror::PoolReplayer: 0x560200de8f30 init: replaying for uuid: bcd54bc5-cd08-435f-a79a-357bce55011d cluster: ceph client: client.mirrorprod 2019-04-05 12:07:29.853633 7f0fa5e284c0 20 rbd::mirror::PoolReplayer: 0x560200de8f30 init: connected to uuid: bcd54bc5-cd08-435f-a79a-357bce55011d cluster: ceph client: client.mirrorprod 2019-04-05 12:07:29.853660 7f0fa5e284c0 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 init: 2019-04-05 12:07:29.853747 7f0f97626700 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 schedule_image_state_check_task: scheduling image state check after 30 sec (task 0x7f0f88000ba0) 2019-04-05 12:07:29.853855 7f0fa5e284c0 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 add_peer: bcd54bc5-cd08-435f-a79a-357bce55011d 2019-04-05 12:07:29.853949 7f0fa5e284c0 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 init: instance_id=464363 2019-04-05 12:07:29.853955 7f0fa5e284c0 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 register_instance: 2019-04-05 12:07:29.859103 7f0f57fff700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 handle_register_instance: r=0 2019-04-05 12:07:29.859125 7f0f57fff700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 create_instance_object: 2019-04-05 12:07:29.866499 7f0f57fff700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 handle_create_instance_object: r=0 2019-04-05 12:07:29.866520 7f0f57fff700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 register_watch: 2019-04-05 12:07:29.869052 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 handle_register_watch: r=0 2019-04-05 12:07:29.869079 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 acquire_lock: 2019-04-05 12:07:29.872993 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 handle_acquire_lock: r=0 2019-04-05 12:07:29.873121 7f0fa5e284c0 20 rbd::mirror::LeaderWatcher: 0x560200fff340 init: notifier_id=464363 2019-04-05 12:07:29.873132 7f0fa5e284c0 20 rbd::mirror::LeaderWatcher: 0x560200fff340 create_leader_object: 2019-04-05 12:07:29.875116 7f0f57fff700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 handle_create_leader_object: r=0 2019-04-05 12:07:29.875129 7f0f57fff700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 register_watch: 2019-04-05 12:07:29.876952 7f0f97626700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 handle_register_watch: r=0 2019-04-05 12:07:29.876964 7f0f97626700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 schedule_acquire_leader_lock: 2019-04-05 12:07:29.876979 7f0f97626700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 schedule_timer_task: scheduling acquire leader lock after 0 sec (task 0x7f0f880010a0) 2019-04-05 12:07:29.877108 7f0f96e25700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 execute_timer_task: 2019-04-05 12:07:29.877115 7f0f96e25700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 is_leader: 0 2019-04-05 12:07:29.877120 7f0f96e25700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 acquire_leader_lock: acquire_attempts=1 2019-04-05 12:07:29.877204 7f0f3b7fe700 20 rbd::mirror::PoolReplayer: 0x560200de8f30 run: enter 2019-04-05 12:07:29.949781 7f0f97626700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 handle_post_acquire_leader_lock: r=0 2019-04-05 12:07:29.949796 7f0f97626700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 init_status_watcher: 2019-04-05 12:07:29.949825 7f0f97626700 20 rbd::mirror::MirrorStatusWatcher: 0x7f0f880014b0 init: 2019-04-05 12:07:29.962723 7f0f57fff700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 handle_init_status_watcher: r=0 2019-04-05 12:07:29.962735 7f0f57fff700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 init_instances: 2019-04-05 12:07:29.962759 7f0f57fff700 20 rbd::mirror::Instances: 0x7f0f34007070 init: 2019-04-05 12:07:29.962761 7f0f57fff700 20 rbd::mirror::Instances: 0x7f0f34007070 get_instances: 2019-04-05 12:07:29.963359 7f0f57fff700 20 rbd::mirror::InstanceWatcher: C_GetInstances: 0x7f0f34007000 finish: r=0 2019-04-05 12:07:29.963378 7f0f57fff700 20 rbd::mirror::Instances: 0x7f0f34007070 handle_get_instances: r=0 2019-04-05 12:07:29.963388 7f0f57fff700 20 rbd::mirror::Instances: 0x7f0f34007070 schedule_remove_task: 2019-04-05 12:07:29.963401 7f0f57fff700 20 rbd::mirror::Instances: 0x7f0f34007070 schedule_remove_task: scheduling instance 464348 remove after 30 sec (task 0x7f0f34007d60) 2019-04-05 12:07:29.963409 7f0f57fff700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 handle_init_instances: r=0 2019-04-05 12:07:29.963411 7f0f57fff700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 notify_listener: 2019-04-05 12:07:29.963413 7f0f57fff700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 is_leader: 1 2019-04-05 12:07:29.963454 7f0f97626700 20 rbd::mirror::PoolReplayer: 0x560200de8f30 handle_post_acquire_leader: 2019-04-05 12:07:29.963466 7f0f97626700 20 rbd::mirror::ServiceDaemon: 0x560200d29bb0 add_or_update_attribute: pool_id=8, key=leader, value=1 2019-04-05 12:07:29.963474 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 handle_acquire_leader: 2019-04-05 12:07:29.963495 7f0f97626700 20 rbd::mirror::ImageSyncThrottler:: 0x7f0f88001140 ImageSyncThrottler: max_concurrent_syncs=5 2019-04-05 12:07:29.963508 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 unsuspend_notify_requests: 2019-04-05 12:07:29.963512 7f0f97626700 20 rbd::mirror::PoolReplayer: 0x560200de8f30 init_local_pool_watcher: 2019-04-05 12:07:29.963521 7f0f97626700 5 rbd::mirror::PoolWatcher: 0x7f0f88001d30 init: 2019-04-05 12:07:29.963524 7f0f97626700 5 rbd::mirror::PoolWatcher: 0x7f0f88001d30 register_watcher: 2019-04-05 12:07:29.965811 7f0f57fff700 5 rbd::mirror::PoolWatcher: 0x7f0f88001d30 handle_register_watcher: r=0 2019-04-05 12:07:29.965824 7f0f57fff700 5 rbd::mirror::PoolWatcher: 0x7f0f88001d30 refresh_images: 2019-04-05 12:07:29.965827 7f0f57fff700 10 rbd::mirror::pool_watcher::RefreshImagesRequest 0x7f0f34006850 mirror_image_list: 2019-04-05 12:07:29.966756 7f0f57fff700 10 rbd::mirror::pool_watcher::RefreshImagesRequest 0x7f0f34006850 handle_mirror_image_list: r=0 2019-04-05 12:07:29.966771 7f0f57fff700 10 rbd::mirror::pool_watcher::RefreshImagesRequest 0x7f0f34006850 finish: r=0 2019-04-05 12:07:29.966775 7f0f57fff700 5 rbd::mirror::PoolWatcher: 0x7f0f88001d30 handle_refresh_images: r=0 2019-04-05 12:07:29.966777 7f0f57fff700 5 rbd::mirror::PoolWatcher: 0x7f0f88001d30 get_mirror_uuid: 2019-04-05 12:07:29.967653 7f0f57fff700 5 rbd::mirror::PoolWatcher: 0x7f0f88001d30 handle_get_mirror_uuid: r=0 2019-04-05 12:07:29.967668 7f0f57fff700 10 rbd::mirror::PoolWatcher: 0x7f0f88001d30 handle_get_mirror_uuid: mirror_uuid=c8c063e4-9a2c-4e13-a9d3-2786a1dbc645 2019-04-05 12:07:29.967673 7f0f57fff700 20 rbd::mirror::PoolWatcher: 0x7f0f88001d30 schedule_listener: 2019-04-05 12:07:29.967718 7f0f97626700 10 rbd::mirror::PoolWatcher: 0x7f0f88001d30 notify_listener: 2019-04-05 12:07:29.967733 7f0f97626700 10 rbd::mirror::PoolReplayer: 0x560200de8f30 handle_update: mirror_uuid=, added_count=0, removed_count=0 2019-04-05 12:07:29.967740 7f0f97626700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 is_leader: 1 2019-04-05 12:07:29.967743 7f0f97626700 20 rbd::mirror::ServiceDaemon: 0x560200d29bb0 add_or_update_attribute: pool_id=8, key=image_local_count, value=0 2019-04-05 12:07:29.967751 7f0f97626700 20 rbd::mirror::PoolReplayer: 0x560200de8f30 operator(): complete handle_update: r=0 2019-04-05 12:07:29.967757 7f0f97626700 20 rbd::mirror::PoolReplayer: 0x560200de8f30 handle_init_local_pool_watcher: r=0 2019-04-05 12:07:29.967761 7f0f97626700 20 rbd::mirror::PoolReplayer: 0x560200de8f30 init_remote_pool_watcher: 2019-04-05 12:07:29.967771 7f0f97626700 5 rbd::mirror::PoolWatcher: 0x7f0f880048a0 init: 2019-04-05 12:07:29.967775 7f0f97626700 5 rbd::mirror::PoolWatcher: 0x7f0f880048a0 register_watcher: 2019-04-05 12:07:29.977576 7f0f3bfff700 5 rbd::mirror::PoolWatcher: 0x7f0f880048a0 handle_register_watcher: r=0 2019-04-05 12:07:29.977588 7f0f3bfff700 5 rbd::mirror::PoolWatcher: 0x7f0f880048a0 refresh_images: 2019-04-05 12:07:29.977591 7f0f3bfff700 10 rbd::mirror::pool_watcher::RefreshImagesRequest 0x7f0f28000ac0 mirror_image_list: 2019-04-05 12:07:29.979217 7f0f3bfff700 10 rbd::mirror::pool_watcher::RefreshImagesRequest 0x7f0f28000ac0 handle_mirror_image_list: r=0 2019-04-05 12:07:29.979252 7f0f3bfff700 10 rbd::mirror::pool_watcher::RefreshImagesRequest 0x7f0f28000ac0 finish: r=0 2019-04-05 12:07:29.979254 7f0f3bfff700 5 rbd::mirror::PoolWatcher: 0x7f0f880048a0 handle_refresh_images: r=0 2019-04-05 12:07:29.979256 7f0f3bfff700 5 rbd::mirror::PoolWatcher: 0x7f0f880048a0 get_mirror_uuid: 2019-04-05 12:07:29.980796 7f0f3bfff700 5 rbd::mirror::PoolWatcher: 0x7f0f880048a0 handle_get_mirror_uuid: r=0 2019-04-05 12:07:29.980827 7f0f3bfff700 10 rbd::mirror::PoolWatcher: 0x7f0f880048a0 handle_get_mirror_uuid: mirror_uuid=8b79b39c-8a08-45c1-9d32-1d87866b036b 2019-04-05 12:07:29.980830 7f0f3bfff700 20 rbd::mirror::PoolWatcher: 0x7f0f880048a0 schedule_listener: 2019-04-05 12:07:29.980871 7f0f97626700 10 rbd::mirror::PoolWatcher: 0x7f0f880048a0 notify_listener: 2019-04-05 12:07:29.980903 7f0f97626700 10 rbd::mirror::PoolReplayer: 0x560200de8f30 handle_update: mirror_uuid=8b79b39c-8a08-45c1-9d32-1d87866b036b, added_count=2, removed_count=0 2019-04-05 12:07:29.980910 7f0f97626700 20 rbd::mirror::LeaderWatcher: 0x560200fff340 is_leader: 1 2019-04-05 12:07:29.980913 7f0f97626700 20 rbd::mirror::ServiceDaemon: 0x560200d29bb0 add_or_update_attribute: pool_id=8, key=image_local_count, value=0 2019-04-05 12:07:29.980919 7f0f97626700 20 rbd::mirror::ServiceDaemon: 0x560200d29bb0 add_or_update_attribute: pool_id=8, key=image_remote_count, value=2 2019-04-05 12:07:29.980928 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 notify_image_acquire: instance_id=464363, global_image_id=06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7 2019-04-05 12:07:29.980934 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 handle_image_acquire: global_image_id=06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7 2019-04-05 12:07:29.980941 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 notify_image_acquire: instance_id=464363, global_image_id=92f46320-d43d-48eb-8a09-b68a1945cc77 2019-04-05 12:07:29.980943 7f0f97626700 20 rbd::mirror::InstanceWatcher: 0x560200ff98e0 handle_image_acquire: global_image_id=92f46320-d43d-48eb-8a09-b68a1945cc77 2019-04-05 12:07:29.980961 7f0f97626700 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 acquire_image: global_image_id=06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7 2019-04-05 12:07:29.980992 7f0f97626700 15 rbd::mirror::ImageReplayer: 0x7f0f8800b1f0 [8/06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7] register_admin_socket_hook: registered asok hook: nfs/06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7 2019-04-05 12:07:29.981132 7f0f97626700 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 acquire_image: 06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7: creating replayer 0x7f0f8800b1f0 2019-04-05 12:07:29.981140 7f0f97626700 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 start_image_replayer: global_image_id=06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7 2019-04-05 12:07:29.981142 7f0f97626700 20 rbd::mirror::ImageReplayer: 0x7f0f8800b1f0 [8/06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7] start: _on_finish_=0 2019-04-05 12:07:29.981152 7f0f97626700 20 rbd::mirror::ImageReplayer: 0x7f0f8800b1f0 [8/06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7] wait_for_deletion: 2019-04-05 12:07:29.981159 7f0f97626700 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 acquire_image: global_image_id=92f46320-d43d-48eb-8a09-b68a1945cc77 2019-04-05 12:07:29.981167 7f0f97626700 15 rbd::mirror::ImageReplayer: 0x7f0f8800d320 [8/92f46320-d43d-48eb-8a09-b68a1945cc77] register_admin_socket_hook: registered asok hook: nfs/92f46320-d43d-48eb-8a09-b68a1945cc77 2019-04-05 12:07:29.981196 7f0f97626700 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 acquire_image: 92f46320-d43d-48eb-8a09-b68a1945cc77: creating replayer 0x7f0f8800d320 2019-04-05 12:07:29.981200 7f0f97626700 20 rbd::mirror::InstanceReplayer: 0x560200ff9350 start_image_replayer: global_image_id=92f46320-d43d-48eb-8a09-b68a1945cc77 2019-04-05 12:07:29.981201 7f0f97626700 20 rbd::mirror::ImageReplayer: 0x7f0f8800d320 [8/92f46320-d43d-48eb-8a09-b68a1945cc77] start: _on_finish_=0 2019-04-05 12:07:29.981203 7f0f97626700 20 rbd::mirror::ImageReplayer: 0x7f0f8800d320 [8/92f46320-d43d-48eb-8a09-b68a1945cc77] wait_for_deletion: |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
