Hello all, I have two separate ceph (10.2.2) clusters and have configured multisite replication between the two. I can see some buckets get synced, however others do not. Both clusters are RHEL7, and I have upgraded libcurl from 7.29 to 7.50 (to avoid http://tracker.ceph.com/issues/15915). Below is some debug output on the 'secondary' zone (bbp-gva-secondary) after uploading a file to the bucket 'bentest1' from onto the master zone (bbp-gva-master). This appears to to be happening very frequently. The size of my bucket pool in the master is ~120GB, however on the secondary site it's only 5GB so things are not very happy at the moment. What steps can I take to work out why RGW cannot create a lock in the log pool? Is there a way to force a full sync, starting fresh (the secondary site is not advertised to users, thus it's okay to even clean pools to start again)? 2016-09-23 09:03:28.498292 7f992e664700 20 execute(): read data: [{"key":6,"val":["bentest1:bbp-gva-master.85732351.16:-1"]}] 2016-09-23 09:03:28.498453 7f992e664700 20 execute(): modified key=bentest1:bbp-gva-master.85732351.16:-1 2016-09-23 09:03:28.498456 7f992e664700 20 wakeup_data_sync_shards: source_zone=bbp-gva-master, shard_ids={6=bentest1:bbp-gva-master.85732351.16:-1} 2016-09-23 09:03:28.498547 7f9a72ffd700 20 incremental_sync(): async update notification: bentest1:bbp-gva-master.85732351.16:-1 2016-09-23 09:03:28.499137 7f9a7dffb700 20 get_system_obj_state: rctx=0x7f9a3c5f8e08 obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 state=0x7f9a0c069848 s->prefetch_data=0 2016-09-23 09:03:28.501379 7f9a72ffd700 20 operate(): sync status for bucket bentest1:bbp-gva-master.85732351.16:-1: 2 2016-09-23 09:03:28.501433 7f9a877fe700 20 reading from .bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 2016-09-23 09:03:28.501447 7f9a877fe700 20 get_system_obj_state: rctx=0x7f9a877fc6d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 state=0x7f9a340cfbe8 s->prefetch_data=0 2016-09-23 09:03:28.503269 7f9a877fe700 20 get_system_obj_state: rctx=0x7f9a877fc6d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 state=0x7f9a340cfbe8 s->prefetch_data=0 2016-09-23 09:03:28.510428 7f9a72ffd700 20 sending request to https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16&format=json&marker=00000000034.4578.3&type=bucket-index&rgwx-zonegroup=bbp-gva 2016-09-23 09:03:28.625755 7f9a72ffd700 20 [inc sync] skipping object: bentest1:bbp-gva-master.85732351.16:-1/1m: non-complete operation 2016-09-23 09:03:28.625759 7f9a72ffd700 20 [inc sync] syncing object: bentest1:bbp-gva-master.85732351.16:-1/1m 2016-09-23 09:03:28.625831 7f9a72ffd700 20 bucket sync single entry (source_zone=bbp-gva-master) b=bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16]):-1/1m[0] log_entry=00000000036.4586.3 op=0 op_state=1 2016-09-23 09:03:28.625857 7f9a72ffd700 5 bucket sync: sync obj: bbp-gva-master/bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16])/1m[0] 2016-09-23 09:03:28.626092 7f9a85ffb700 20 get_obj_state: rctx=0x7f9a85ff96a0 obj=bentest1:1m state=0x7f9a30051cf8 s->prefetch_data=0 2016-09-23 09:03:28.626119 7f9a72ffd700 20 sending request to https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16&format=json&marker=00000000036.4586.3&type=bucket-index&rgwx-zonegroup=bbp-gva 2016-09-23 09:03:28.627560 7f9a85ffb700 10 get_canon_resource(): dest=/bentest1/1m /bentest1/1m 2016-09-23 09:03:28.627612 7f9a85ffb700 20 sending request to https://bbpobjectstorage.epfl.ch:443/bentest1/1m?rgwx-zonegroup=bbp-gva&rgwx-prepend-metadata=bbp-gva 2016-09-23 09:03:28.725185 7f9a72ffd700 20 incremental_sync:1067: shard_id=6 log_entry: 1_1474614207.373384_1713810.1:2016-09-23 09:03:27.0.373384s:bentest1:bbp-gva-master.85732351.16 2016-09-23 09:03:28.725477 7f9a9affd700 20 get_system_obj_state: rctx=0x7f9a3c5f8e08 obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 state=0x7f9a741bb0a8 s->prefetch_data=0 2016-09-23 09:03:28.728404 7f9a72ffd700 20 operate(): sync status for bucket bentest1:bbp-gva-master.85732351.16:-1: 2 2016-09-23 09:03:28.728462 7f9a7b7f6700 20 reading from .bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 2016-09-23 09:03:28.728490 7f9a7b7f6700 20 get_system_obj_state: rctx=0x7f9a7b7f46d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 state=0x7f9a000b19b8 s->prefetch_data=0 2016-09-23 09:03:28.729664 7f9a7b7f6700 20 get_system_obj_state: rctx=0x7f9a7b7f46d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 state=0x7f9a000b19b8 s->prefetch_data=0 2016-09-23 09:03:28.731703 7f9a72ffd700 20 cr:s=0x7f9a3c5a4f90:op=0x7f9a3ca75ef0:20RGWContinuousLeaseCR: couldn't lock .bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16:sync_lock: retcode=-16 2016-09-23 09:03:28.731721 7f9a72ffd700 0 ERROR: incremental sync on bentest1 bucket_id=bbp-gva-master.85732351.16 shard_id=-1 failed, retcode=-16 2016-09-23 09:03:28.758421 7f9a72ffd700 20 store_marker(): updating marker marker_oid=bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 marker=00000000035.4585.2 2016-09-23 09:03:28.829207 7f9a72ffd700 0 ERROR: failed to sync object: bentest1:bbp-gva-master.85732351.16:-1/1m 2016-09-23 09:03:28.834281 7f9a72ffd700 20 store_marker(): updating marker marker_oid=bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 marker=00000000036.4586.3 -- Kind regards, Ben Morrice ______________________________________________________________________ Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670 EPFL ENT CBS BBP Biotech Campus Chemin des Mines 9 1202 Geneva Switzerland _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |