Hi Ben, On Mon, Apr 24, 2017 at 4:36 PM, Ben Morrice <ben.morrice@xxxxxxx> wrote: > Hello Orit, > > Could it be that something has changed in 10.2.5+ which is related to > reading the endpoints from the zone/period config? > I don't remember any change for endpoints config, but I will go over the changes to make sure. There were a few changes with tenant handling that may cause this regression. > In my master zone I have specified the endpoint with a trailing backslash > (which is also escaped), however I do not define the secondary endpoint this > way. Am I hitting a bug here? > Can you update the secondary endpoint and see if it helps? Please open a bug in tracker with regarding to this issue. Regards, Orit > Kind regards, > > Ben Morrice > > ______________________________________________________________________ > Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670 > EPFL / BBP > Biotech Campus > Chemin des Mines 9 > 1202 Geneva > Switzerland > > On 21/04/17 09:36, Ben Morrice wrote: >> >> Hello Orit, >> >> Please find attached the output from the radosgw commands and the relevant >> section from ceph.conf (radosgw) >> >> bbp-gva-master is running 10.2.5 >> >> bbp-gva-secondary is running 10.2.7 >> >> Kind regards, >> >> Ben Morrice >> >> ______________________________________________________________________ >> Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670 >> EPFL / BBP >> Biotech Campus >> Chemin des Mines 9 >> 1202 Geneva >> Switzerland >> >> On 21/04/17 07:55, Orit Wasserman wrote: >>> >>> Hi Ben, >>> >>> On Thu, Apr 20, 2017 at 6:08 PM, Ben Morrice <ben.morrice@xxxxxxx> wrote: >>>> >>>> Hi all, >>>> >>>> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 >>>> (RHEL7) >>>> and authentication is in a very bad state. This installation is part of >>>> a >>>> multigw configuration, and I have just updated one host in the secondary >>>> zone (all other hosts/zones are running 10.2.5). >>>> >>>> On the 10.2.7 server I cannot authenticate as a user (normally backed by >>>> OpenStack Keystone), but even worse I can also not authenticate with an >>>> admin user. >>>> >>>> Please see [1] for the results of performing a list bucket operation >>>> with >>>> python boto (script works against rgw 10.2.5) >>>> >>>> Also, if I try to authenticate from the 'master' rgw zone with a >>>> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get: >>>> >>>> "ERROR: failed to fetch datalog info" >>>> >>>> "failed to retrieve sync info: (13) Permission denied" >>>> >>>> The above errors correlates to the errors in the log on the server >>>> running >>>> 10.2.7 (debug level 20) at [2] >>>> >>>> I'm not sure what I have done wrong or can try next? >>>> >>>> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns >>>> authentication functionality >>> >>> Can you provide the following info: >>> radosgw-admin period get >>> radsogw-admin zonegroup get >>> radsogw-admin zone get >>> >>> Can you provide your ceph.conf? >>> >>> Thanks, >>> Orit >>> >>>> [1] >>>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden >>>> <?xml version="1.0" >>>> >>>> encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000004-0058f8c86a-3fa2959-bbp-gva-secondary</RequestId><HostId>3fa2959-bbp-gva-secondary-bbp-gva</HostId></Error> >>>> >>>> [2] >>>> /bbpsrvc15.cscs.ch/admin/log >>>> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated >>>> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM= >>>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 >>>> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU= >>>> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34 >>>> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request >>>> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER: >>>> err_no=-2027 new_err_no=-2027 >>>> 2017-04-20 16:43:04.916329 7ff87c6c0700 2 req 354:0.052585:s3:GET >>>> /admin/log:get_obj:op status=0 >>>> 2017-04-20 16:43:04.916339 7ff87c6c0700 2 req 354:0.052595:s3:GET >>>> /admin/log:get_obj:http status=403 >>>> 2017-04-20 16:43:04.916343 7ff87c6c0700 1 ====== req done >>>> req=0x7ff87c6ba710 op status=0 http_status=403 ====== >>>> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned >>>> -2027 >>>> 2017-04-20 16:43:04.916390 7ff87c6c0700 1 civetweb: 0x7ff990015610: >>>> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" >>>> 403 0 >>>> - - >>>> 2017-04-20 16:43:04.917212 7ff9777e6700 20 >>>> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate() >>>> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync: >>>> incremental_sync:1544: shard_id=20 >>>> mdlog_marker=1_1492686039.901886_5551978.1 >>>> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker= >>>> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync: >>>> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20 >>>> 2017-04-20 16:43:04.917236 7ff9777e6700 20 >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: >>>> operate() >>>> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate: >>>> shard_id=20: init request >>>> 2017-04-20 16:43:04.917240 7ff9777e6700 20 >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: >>>> operate() >>>> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate: >>>> shard_id=20: reading shard status >>>> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is >>>> io >>>> blocked >>>> 2017-04-20 16:43:04.918285 7ff9777e6700 20 >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: >>>> operate() >>>> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate: >>>> shard_id=20: reading shard status complete >>>> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20 >>>> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20 >>>> 13:00:39.0.901886s >>>> 2017-04-20 16:43:04.918316 7ff9777e6700 20 >>>> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: >>>> operate() >>>> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate: >>>> shard_id=20: sending rest request >>>> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu >>>> Apr >>>> 20 14:43:04 2017 >>>> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20 >>>> 14:43:04 2017 >>>> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource(): >>>> dest=/admin/log >>>> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: >>>> GET >>>> >>>> -- >>>> Kind regards, >>>> >>>> Ben Morrice >>>> >>>> ______________________________________________________________________ >>>> Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670 >>>> EPFL / BBP >>>> Biotech Campus >>>> Chemin des Mines 9 >>>> 1202 Geneva >>>> Switzerland >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |