Hello Orit, Thanks for your help so far. The bug you referenced was not included in 10.2.3. I cherry-picked the commits mentioned in http://tracker.ceph.com/issues/16742 into the 10.2.3 release and deployed this radosgw on the servers affected. Unfortunately it's still failing, now the sync and subsequent retries of the sync fail with a return code of -5. Any other suggestions? 2016-09-28 16:14:52.145933 7f84609e3700 20 rgw meta sync: entry: name=20160928:bbp-gva-master.106061599.1 2016-09-28 16:14:52.145994 7f84609e3700 20 rgw meta sync: entry: name=20160928:bbp-gva-master.106061599.1 2016-09-28 16:14:52.145998 7f84609e3700 20 rgw meta sync: entry: name=20160928 2016-09-28 16:14:52.146001 7f84609e3700 20 rgw meta sync: entry: name=20160928 2016-09-28 16:14:52.151393 7f84609e3700 20 rgw meta sync: incremental_sync:1576: shard_id=11 log_entry: 1_1475072090.900125_1286081.1:bucket.instance:20160928:bbp-gva-master.106061599.1:2016-09-28 16:14:50.900125 2016-09-28 16:14:52.151507 7f84609e3700 20 rgw meta sync: incremental_sync:1576: shard_id=11 log_entry: 1_1475072090.914533_1286082.1:bucket.instance:20160928:bbp-gva-master.106061599.1:2016-09-28 16:14:50.914533 2016-09-28 16:14:52.151524 7f84609e3700 20 rgw meta sync: fetching remote metadata: bucket.instance:20160928:bbp-gva-master.106061599.1 2016-09-28 16:14:52.151533 7f84609e3700 20 rgw meta sync: incremental_sync:1576: shard_id=11 log_entry: 1_1475072090.918249_1286083.1:bucket:20160928:2016-09-28 16:14:50.918249 2016-09-28 16:14:52.151700 7f84609e3700 10 get_canon_resource(): dest=/admin/metadata/bucket.instance/20160928:bbp-gva-master.106061599.1 /admin/metadata/bucket.instance/20160928:bbp-gva-master.106061599.1 2016-09-28 16:14:52.151756 7f84609e3700 20 sending request to https://bbpobjectstorage.epfl.ch:443/admin/metadata/bucket.instance/20160928:bbp-gva-master.106061599.1?key=20160928%3Abbp-gva-master.106061599.1&rgwx-zonegroup=bbp-gva 2016-09-28 16:14:52.151814 7f84609e3700 20 rgw meta sync: incremental_sync:1576: shard_id=11 log_entry: 1_1475072090.933082_1286084.1:bucket:20160928:2016-09-28 16:14:50.933082 2016-09-28 16:14:52.151839 7f84609e3700 20 rgw meta sync: fetching remote metadata: bucket:20160928 2016-09-28 16:14:52.152030 7f84609e3700 10 get_canon_resource(): dest=/admin/metadata/bucket/20160928 /admin/metadata/bucket/20160928 2016-09-28 16:14:52.152086 7f84609e3700 20 sending request to https://bbpobjectstorage.epfl.ch:443/admin/metadata/bucket/20160928?key=20160928&rgwx-zonegroup=bbp-gva 2016-09-28 16:14:52.299619 7f8471ffb700 20 get_system_obj_state: rctx=0x7f8471ff9200 obj=.bbp-gva-secondary.domain.rgw:20160928 state=0x7f842c24b7f8 s->prefetch_data=0 2016-09-28 16:14:52.398943 7f846a7fc700 20 reading from .bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 2016-09-28 16:14:52.398975 7f846a7fc700 20 get_system_obj_state: rctx=0x7f846a7fa150 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f8420de2af8 s->prefetch_data=0 2016-09-28 16:15:38.111569 7f828ed65700 20 execute(): read data: [{"key":16,"val":["20160928:bbp-gva-master.106061599.1"]}] 2016-09-28 16:15:38.111836 7f828ed65700 20 execute(): modified key=20160928:bbp-gva-master.106061599.1 2016-09-28 16:15:38.111839 7f828ed65700 20 wakeup_data_sync_shards: source_zone=bbp-gva-master, shard_ids={16=20160928:bbp-gva-master.106061599.1} 2016-09-28 16:15:38.111919 7f845b7fe700 20 incremental_sync(): async update notification: 20160928:bbp-gva-master.106061599.1 2016-09-28 16:15:38.112248 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:15:38.112661 7f8473fff700 20 get_system_obj_state: rctx=0x7f84283c17f8 obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:20160928:bbp-gva-master.106061599.1 state=0x7f844405b338 s->prefetch_data=0 2016-09-28 16:15:38.115079 7f845b7fe700 20 operate(): sync status for bucket 20160928:bbp-gva-master.106061599.1: 0 2016-09-28 16:15:38.115202 7f8491bf7700 20 reading from .bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 2016-09-28 16:15:38.115232 7f8491bf7700 20 get_system_obj_state: rctx=0x7f8491bf56d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f84540808d8 s->prefetch_data=0 2016-09-28 16:15:38.118544 7f8491bf7700 20 get_system_obj_state: rctx=0x7f8491bf56d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f84540808d8 s->prefetch_data=0 2016-09-28 16:15:38.138092 7f845b7fe700 20 sending request to https://bbpobjectstorage.epfl.ch:443/admin/log/?type=bucket-index&bucket-instance=20160928%3Abbp-gva-master.106061599.1&info&rgwx-zonegroup=bbp-gva 2016-09-28 16:15:38.223131 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:15:38.227125 7f845b7fe700 10 get_canon_resource(): dest=/20160928?versions /20160928?versions 2016-09-28 16:15:38.227201 7f845b7fe700 20 sending request to https://bbpobjectstorage.epfl.ch:443/20160928?rgwx-bucket-instance=20160928%3Abbp-gva-master.106061599.1&versions&format=json&objs-container=true&key-marker&version-id-marker&rgwx-zonegroup=bbp-gva 2016-09-28 16:15:38.273996 7f845b7fe700 20 incremental_sync:1228: shard_id=16 log_entry: 1_1475072137.002597_1324310.1:2016-09-28 16:15:37.0.002597s:20160928:bbp-gva-master.106061599.1 2016-09-28 16:15:38.274169 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:15:38.274480 7f846affd700 20 get_system_obj_state: rctx=0x7f84283bb4c8 obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:20160928:bbp-gva-master.106061599.1 state=0x7f841c3f3d98 s->prefetch_data=0 2016-09-28 16:15:38.276880 7f845b7fe700 20 operate(): sync status for bucket 20160928:bbp-gva-master.106061599.1: 1 2016-09-28 16:15:38.276990 7f84677f6700 20 reading from .bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 2016-09-28 16:15:38.277008 7f84677f6700 20 get_system_obj_state: rctx=0x7f84677f46d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f8408166968 s->prefetch_data=0 2016-09-28 16:15:38.278272 7f84677f6700 20 get_system_obj_state: rctx=0x7f84677f46d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f8408166968 s->prefetch_data=0 2016-09-28 16:15:38.280852 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:15:38.282199 7f845b7fe700 20 cr:s=0x7f84282e6640:op=0x7f842832df70:20RGWContinuousLeaseCR: couldn't lock .bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:20160928:bbp-gva-master.106061599.1:sync_lock: retcode=-16 2016-09-28 16:15:38.282216 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:finish 2016-09-28 16:15:38.282221 7f845b7fe700 5 full sync on 20160928:bbp-gva-master.106061599.1 failed, retcode=-16 2016-09-28 16:15:38.282225 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:finish 2016-09-28 16:15:38.309639 7f845b7fe700 20 [full sync] syncing object: 20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null] 2016-09-28 16:15:38.309713 7f845b7fe700 20 bucket sync single entry (source_zone=bbp-gva-master) b=20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0] log_entry=20160928-1mb-testfile[null] op=0 op_state=1 2016-09-28 16:15:38.309723 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:start 2016-09-28 16:15:38.309739 7f845b7fe700 5 bucket sync: sync obj: bbp-gva-master/20160928(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-secondary.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.106061599.1])/20160928-1mb-testfile[null][0] 2016-09-28 16:15:38.309745 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:fetch 2016-09-28 16:15:38.310004 7f84627ec700 20 get_obj_state: rctx=0x7f84627ea6a0 obj=20160928:20160928-1mb-testfile state=0x7f851ee59e28 s->prefetch_data=0 2016-09-28 16:15:38.311467 7f84627ec700 10 get_canon_resource(): dest=/20160928/20160928-1mb-testfile?versionId=null /20160928/20160928-1mb-testfile?versionId=null 2016-09-28 16:15:38.311562 7f84627ec700 20 sending request to https://bbpobjectstorage.epfl.ch:443/20160928/20160928-1mb-testfile?rgwx-zonegroup=bbp-gva&rgwx-prepend-metadata=bbp-gva&versionId=null 2016-09-28 16:15:38.393174 7f84627ec700 10 received header:x-amz-meta-orig-filename: 20160928-1mb-testfile 2016-09-28 16:15:38.467281 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:done, retcode=-5 2016-09-28 16:15:38.467289 7f845b7fe700 0 ERROR: failed to sync object: 20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile 2016-09-28 16:15:38.473062 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:finish 2016-09-28 16:15:38.480412 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:finish 2016-09-28 16:15:38.480432 7f845b7fe700 5 full sync on 20160928:bbp-gva-master.106061599.1 failed, retcode=-5 2016-09-28 16:15:38.480436 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:finish 2016-09-28 16:15:40.657736 7f845b7fe700 20 incremental_sync(): handle error entry: 20160928:bbp-gva-master.106061599.1 2016-09-28 16:15:40.657791 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:15:40.658382 7f84657f2700 20 get_system_obj_state: rctx=0x7f84283ac478 obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:20160928:bbp-gva-master.106061599.1 state=0x7f83ec030e88 s->prefetch_data=0 2016-09-28 16:15:40.661356 7f845b7fe700 20 operate(): sync status for bucket 20160928:bbp-gva-master.106061599.1: 1 2016-09-28 16:15:40.661411 7f8471ffb700 20 reading from .bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 2016-09-28 16:15:40.661431 7f8471ffb700 20 get_system_obj_state: rctx=0x7f8471ff96d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f842c24b7f8 s->prefetch_data=0 2016-09-28 16:15:40.662709 7f8471ffb700 20 get_system_obj_state: rctx=0x7f8471ff96d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f842c24b7f8 s->prefetch_data=0 2016-09-28 16:15:40.663968 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:15:40.667246 7f845b7fe700 10 get_canon_resource(): dest=/20160928?versions /20160928?versions 2016-09-28 16:15:40.667318 7f845b7fe700 20 sending request to https://bbpobjectstorage.epfl.ch:443/20160928?rgwx-bucket-instance=20160928%3Abbp-gva-master.106061599.1&versions&format=json&objs-container=true&key-marker&version-id-marker&rgwx-zonegroup=bbp-gva 2016-09-28 16:15:40.861206 7f845b7fe700 20 [full sync] syncing object: 20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null] 2016-09-28 16:15:40.861259 7f845b7fe700 20 bucket sync single entry (source_zone=bbp-gva-master) b=20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0] log_entry=20160928-1mb-testfile[null] op=0 op_state=1 2016-09-28 16:15:40.861268 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:start 2016-09-28 16:15:40.861282 7f845b7fe700 5 bucket sync: sync obj: bbp-gva-master/20160928(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-secondary.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.106061599.1])/20160928-1mb-testfile[null][0] 2016-09-28 16:15:40.861288 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:fetch 2016-09-28 16:15:40.861501 7f8469ffb700 20 get_obj_state: rctx=0x7f8469ff96a0 obj=20160928:20160928-1mb-testfile state=0x7f8414075228 s->prefetch_data=0 2016-09-28 16:15:40.862592 7f8469ffb700 10 get_canon_resource(): dest=/20160928/20160928-1mb-testfile?versionId=null /20160928/20160928-1mb-testfile?versionId=null 2016-09-28 16:15:40.862640 7f8469ffb700 20 sending request to https://bbpobjectstorage.epfl.ch:443/20160928/20160928-1mb-testfile?rgwx-zonegroup=bbp-gva&rgwx-prepend-metadata=bbp-gva&versionId=null 2016-09-28 16:15:40.993982 7f8469ffb700 10 received header:x-amz-meta-orig-filename: 20160928-1mb-testfile 2016-09-28 16:15:41.078155 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:done, retcode=-5 2016-09-28 16:15:41.078159 7f845b7fe700 0 ERROR: failed to sync object: 20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile 2016-09-28 16:15:41.082012 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:finish 2016-09-28 16:15:41.084894 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:finish 2016-09-28 16:15:41.084912 7f845b7fe700 5 full sync on 20160928:bbp-gva-master.106061599.1 failed, retcode=-5 2016-09-28 16:15:41.084916 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:finish 2016-09-28 16:19:01.870803 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:19:01.871103 7f84647f0700 20 get_system_obj_state: rctx=0x7f8428e20138 obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:20160928:bbp-gva-master.106061599.1 state=0x7f83e4216d08 s->prefetch_data=0 2016-09-28 16:19:01.873644 7f845b7fe700 20 operate(): sync status for bucket 20160928:bbp-gva-master.106061599.1: 1 2016-09-28 16:19:01.873680 7f84657f2700 20 reading from .bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 2016-09-28 16:19:01.873695 7f84657f2700 20 get_system_obj_state: rctx=0x7f84657f06d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f83ec04c598 s->prefetch_data=0 2016-09-28 16:19:01.874946 7f84657f2700 20 get_system_obj_state: rctx=0x7f84657f06d0 obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.20160928:bbp-gva-master.106061599.1 state=0x7f83ec04c598 s->prefetch_data=0 2016-09-28 16:19:01.876348 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:start 2016-09-28 16:19:01.879886 7f845b7fe700 10 get_canon_resource(): dest=/20160928?versions /20160928?versions 2016-09-28 16:19:01.879930 7f845b7fe700 20 sending request to https://bbpobjectstorage.epfl.ch:443/20160928?rgwx-bucket-instance=20160928%3Abbp-gva-master.106061599.1&versions&format=json&objs-container=true&key-marker&version-id-marker&rgwx-zonegroup=bbp-gva 2016-09-28 16:19:01.968906 7f845b7fe700 20 [full sync] syncing object: 20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null] 2016-09-28 16:19:01.968988 7f845b7fe700 20 bucket sync single entry (source_zone=bbp-gva-master) b=20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0] log_entry=20160928-1mb-testfile[null] op=0 op_state=1 2016-09-28 16:19:01.969003 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:start 2016-09-28 16:19:01.969014 7f845b7fe700 5 bucket sync: sync obj: bbp-gva-master/20160928(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-secondary.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.106061599.1])/20160928-1mb-testfile[null][0] 2016-09-28 16:19:01.969017 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:fetch 2016-09-28 16:19:01.969363 7f84913f6700 20 get_obj_state: rctx=0x7f84913f46a0 obj=20160928:20160928-1mb-testfile state=0x7f844c17f348 s->prefetch_data=0 2016-09-28 16:19:01.970699 7f84913f6700 10 get_canon_resource(): dest=/20160928/20160928-1mb-testfile?versionId=null /20160928/20160928-1mb-testfile?versionId=null 2016-09-28 16:19:01.970882 7f84913f6700 20 sending request to https://bbpobjectstorage.epfl.ch:443/20160928/20160928-1mb-testfile?rgwx-zonegroup=bbp-gva&rgwx-prepend-metadata=bbp-gva&versionId=null 2016-09-28 16:19:02.087169 7f84913f6700 10 received header:x-amz-meta-orig-filename: 20160928-1mb-testfile 2016-09-28 16:19:02.156463 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:done, retcode=-5 2016-09-28 16:19:02.156467 7f845b7fe700 0 ERROR: failed to sync object: 20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile 2016-09-28 16:19:02.160115 7f845b7fe700 5 Sync:bbp-gva-:data:Object:20160928:bbp-gva-master.106061599.1/20160928-1mb-testfile[null][0]:finish 2016-09-28 16:19:02.163101 7f845b7fe700 5 Sync:bbp-gva-:data:BucketFull:20160928:bbp-gva-master.106061599.1:finish 2016-09-28 16:19:02.163108 7f845b7fe700 5 full sync on 20160928:bbp-gva-master.106061599.1 failed, retcode=-5 2016-09-28 16:19:02.163111 7f845b7fe700 5 Sync:bbp-gva-:data:Bucket:20160928:bbp-gva-master.106061599.1:finish Kind regards, Ben Morrice ______________________________________________________________________ Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670 EPFL ENT CBS BBP Biotech Campus Chemin des Mines 9 1202 Geneva Switzerland On 27/09/16 09:36, Ben Morrice wrote: > Hello Orit, > > Yes, this bug looks to correlate. Was this included in 10.2.3? > > I guess not as I have since updated to 10.2.3 but getting the same errors > > This bug talks about not retrying after a failure, however do you know > why the sync fails in the first place? It seems that basically any > object over 500k in size fails :( > > Kind regards, > > Ben Morrice > > ______________________________________________________________________ > Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670 > EPFL ENT CBS BBP > Biotech Campus > Chemin des Mines 9 > 1202 Geneva > Switzerland > > On 23/09/16 16:52, Orit Wasserman wrote: >> Hi Ben, >> It seems to be http://tracker.ceph.com/issues/16742. >> It is being backported to jewel http://tracker.ceph.com/issues/16794, >> you can try apply it and see if it helps you. >> >> Regards, >> Orit >> >> On Fri, Sep 23, 2016 at 9:21 AM, Ben Morrice <ben.morrice@xxxxxxx> wrote: >>> Hello all, >>> >>> I have two separate ceph (10.2.2) clusters and have configured multisite >>> replication between the two. I can see some buckets get synced, however >>> others do not. >>> >>> Both clusters are RHEL7, and I have upgraded libcurl from 7.29 to 7.50 >>> (to avoid http://tracker.ceph.com/issues/15915). >>> >>> Below is some debug output on the 'secondary' zone (bbp-gva-secondary) >>> after uploading a file to the bucket 'bentest1' from onto the master >>> zone (bbp-gva-master). >>> >>> This appears to to be happening very frequently. The size of my bucket >>> pool in the master is ~120GB, however on the secondary site it's only >>> 5GB so things are not very happy at the moment. >>> >>> What steps can I take to work out why RGW cannot create a lock in the >>> log pool? >>> >>> Is there a way to force a full sync, starting fresh (the secondary site >>> is not advertised to users, thus it's okay to even clean pools to start >>> again)? >>> >>> >>> 2016-09-23 09:03:28.498292 7f992e664700 20 execute(): read data: >>> [{"key":6,"val":["bentest1:bbp-gva-master.85732351.16:-1"]}] >>> 2016-09-23 09:03:28.498453 7f992e664700 20 execute(): modified >>> key=bentest1:bbp-gva-master.85732351.16:-1 >>> 2016-09-23 09:03:28.498456 7f992e664700 20 wakeup_data_sync_shards: >>> source_zone=bbp-gva-master, >>> shard_ids={6=bentest1:bbp-gva-master.85732351.16:-1} >>> 2016-09-23 09:03:28.498547 7f9a72ffd700 20 incremental_sync(): async >>> update notification: bentest1:bbp-gva-master.85732351.16:-1 >>> 2016-09-23 09:03:28.499137 7f9a7dffb700 20 get_system_obj_state: >>> rctx=0x7f9a3c5f8e08 >>> obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 >>> state=0x7f9a0c069848 s->prefetch_data=0 >>> 2016-09-23 09:03:28.501379 7f9a72ffd700 20 operate(): sync status for >>> bucket bentest1:bbp-gva-master.85732351.16:-1: 2 >>> 2016-09-23 09:03:28.501433 7f9a877fe700 20 reading from >>> .bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 >>> 2016-09-23 09:03:28.501447 7f9a877fe700 20 get_system_obj_state: >>> rctx=0x7f9a877fc6d0 >>> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 >>> state=0x7f9a340cfbe8 s->prefetch_data=0 >>> 2016-09-23 09:03:28.503269 7f9a877fe700 20 get_system_obj_state: >>> rctx=0x7f9a877fc6d0 >>> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 >>> state=0x7f9a340cfbe8 s->prefetch_data=0 >>> 2016-09-23 09:03:28.510428 7f9a72ffd700 20 sending request to >>> https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16&format=json&marker=00000000034.4578.3&type=bucket-index&rgwx-zonegroup=bbp-gva >>> 2016-09-23 09:03:28.625755 7f9a72ffd700 20 [inc sync] skipping object: >>> bentest1:bbp-gva-master.85732351.16:-1/1m: non-complete operation >>> 2016-09-23 09:03:28.625759 7f9a72ffd700 20 [inc sync] syncing object: >>> bentest1:bbp-gva-master.85732351.16:-1/1m >>> 2016-09-23 09:03:28.625831 7f9a72ffd700 20 bucket sync single entry >>> (source_zone=bbp-gva-master) >>> b=bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16]):-1/1m[0] >>> log_entry=00000000036.4586.3 op=0 op_state=1 >>> 2016-09-23 09:03:28.625857 7f9a72ffd700 5 bucket sync: sync obj: >>> bbp-gva-master/bentest1(@{i=.bbp-gva-secondary.rgw.buckets.index,e=.bbp-gva-master.rgw.buckets.extra}.bbp-gva-secondary.rgw.buckets[bbp-gva-master.85732351.16])/1m[0] >>> 2016-09-23 09:03:28.626092 7f9a85ffb700 20 get_obj_state: >>> rctx=0x7f9a85ff96a0 obj=bentest1:1m state=0x7f9a30051cf8 s->prefetch_data=0 >>> 2016-09-23 09:03:28.626119 7f9a72ffd700 20 sending request to >>> https://bbpobjectstorage.epfl.ch:443/admin/log?bucket-instance=bentest1%3Abbp-gva-master.85732351.16&format=json&marker=00000000036.4586.3&type=bucket-index&rgwx-zonegroup=bbp-gva >>> 2016-09-23 09:03:28.627560 7f9a85ffb700 10 get_canon_resource(): >>> dest=/bentest1/1m >>> /bentest1/1m >>> 2016-09-23 09:03:28.627612 7f9a85ffb700 20 sending request to >>> https://bbpobjectstorage.epfl.ch:443/bentest1/1m?rgwx-zonegroup=bbp-gva&rgwx-prepend-metadata=bbp-gva >>> 2016-09-23 09:03:28.725185 7f9a72ffd700 20 incremental_sync:1067: >>> shard_id=6 log_entry: 1_1474614207.373384_1713810.1:2016-09-23 >>> 09:03:27.0.373384s:bentest1:bbp-gva-master.85732351.16 >>> 2016-09-23 09:03:28.725477 7f9a9affd700 20 get_system_obj_state: >>> rctx=0x7f9a3c5f8e08 >>> obj=.bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 >>> state=0x7f9a741bb0a8 s->prefetch_data=0 >>> 2016-09-23 09:03:28.728404 7f9a72ffd700 20 operate(): sync status for >>> bucket bentest1:bbp-gva-master.85732351.16:-1: 2 >>> 2016-09-23 09:03:28.728462 7f9a7b7f6700 20 reading from >>> .bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 >>> 2016-09-23 09:03:28.728490 7f9a7b7f6700 20 get_system_obj_state: >>> rctx=0x7f9a7b7f46d0 >>> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 >>> state=0x7f9a000b19b8 s->prefetch_data=0 >>> 2016-09-23 09:03:28.729664 7f9a7b7f6700 20 get_system_obj_state: >>> rctx=0x7f9a7b7f46d0 >>> obj=.bbp-gva-secondary.domain.rgw:.bucket.meta.bentest1:bbp-gva-master.85732351.16 >>> state=0x7f9a000b19b8 s->prefetch_data=0 >>> 2016-09-23 09:03:28.731703 7f9a72ffd700 20 >>> cr:s=0x7f9a3c5a4f90:op=0x7f9a3ca75ef0:20RGWContinuousLeaseCR: couldn't >>> lock >>> .bbp-gva-secondary.log:bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16:sync_lock: >>> retcode=-16 >>> 2016-09-23 09:03:28.731721 7f9a72ffd700 0 ERROR: incremental sync on >>> bentest1 bucket_id=bbp-gva-master.85732351.16 shard_id=-1 failed, >>> retcode=-16 >>> 2016-09-23 09:03:28.758421 7f9a72ffd700 20 store_marker(): updating >>> marker >>> marker_oid=bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 >>> marker=00000000035.4585.2 >>> 2016-09-23 09:03:28.829207 7f9a72ffd700 0 ERROR: failed to sync object: >>> bentest1:bbp-gva-master.85732351.16:-1/1m >>> 2016-09-23 09:03:28.834281 7f9a72ffd700 20 store_marker(): updating >>> marker >>> marker_oid=bucket.sync-status.bbp-gva-master:bentest1:bbp-gva-master.85732351.16 >>> marker=00000000036.4586.3 >>> >>> >>> >>> -- >>> Kind regards, >>> >>> Ben Morrice >>> >>> ______________________________________________________________________ >>> Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670 >>> EPFL ENT CBS BBP >>> Biotech Campus >>> Chemin des Mines 9 >>> 1202 Geneva >>> Switzerland >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |