Hello Ben,
Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("====== starting new request...") till the end marker?
At the moment we can't see any details related to the signature
calculation.
Regards,
Radek
On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morrice@xxxxxxx> wrote:
> Hi all,
>
> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
> and authentication is in a very bad state. This installation is part of a
> multigw configuration, and I have just updated one host in the secondary
> zone (all other hosts/zones are running 10.2.5).
>
> On the 10.2.7 server I cannot authenticate as a user (normally backed by
> OpenStack Keystone), but even worse I can also not authenticate with an
> admin user.
>
> Please see [1] for the results of performing a list bucket operation with
> python boto (script works against rgw 10.2.5)
>
> Also, if I try to authenticate from the 'master' rgw zone with a
> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>
> "ERROR: failed to fetch datalog info"
>
> "failed to retrieve sync info: (13) Permission denied"
>
> The above errors correlates to the errors in the log on the server running
> 10.2.7 (debug level 20) at [2]
>
> I'm not sure what I have done wrong or can try next?
>
> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
> authentication functionality
>
> [1]
> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
> <?xml version="1.0"
> encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000004-0058f8c86a-3fa2959-bbp-gva-secondary</RequestId><HostId>3fa2959-bbp-gva-secondary-bbp-gva</HostId></Error>
>
> [2]
> /bbpsrvc15.cscs.ch/admin/log
> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
> err_no=-2027 new_err_no=-2027
> 2017-04-20 16:43:04.916329 7ff87c6c0700 2 req 354:0.052585:s3:GET
> /admin/log:get_obj:op status=0
> 2017-04-20 16:43:04.916339 7ff87c6c0700 2 req 354:0.052595:s3:GET
> /admin/log:get_obj:http status=403
> 2017-04-20 16:43:04.916343 7ff87c6c0700 1 ====== req done
> req=0x7ff87c6ba710 op status=0 http_status=403 ======
> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
> 2017-04-20 16:43:04.916390 7ff87c6c0700 1 civetweb: 0x7ff990015610:
> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
> - -
> 2017-04-20 16:43:04.917212 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1544: shard_id=20
> mdlog_marker=1_1492686039.901886_5551978.1
> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
> 2017-04-20 16:43:04.917236 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: init request
> 2017-04-20 16:43:04.917240 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status
> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
> blocked
> 2017-04-20 16:43:04.918285 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status complete
> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
> 13:00:39.0.901886s
> 2017-04-20 16:43:04.918316 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: sending rest request
> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
> 20 14:43:04 2017
> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
> 14:43:04 2017
> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
> dest=/admin/log
> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET
>
> --
> Kind regards,
>
> Ben Morrice
>
> ______________________________________________________________________
> Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
