Hello Radek,
Please find attached the failed request for both the admin user and a standard user (backed by keystone).
Kind regards,
Ben Morrice
______________________________________________________________________
Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland
________________________________________
From: Radoslaw Zarzynski <rzarzynski@xxxxxxxxxxxx>
Sent: Tuesday, April 25, 2017 7:38 PM
To: Morrice Ben
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: RGW 10.2.5->10.2.7 authentication fail?
Hello Ben,
Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("====== starting new request...") till the end marker?
At the moment we can't see any details related to the signature
calculation.
Regards,
Radek
On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morrice@xxxxxxx> wrote:
> Hi all,
>
> I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
> and authentication is in a very bad state. This installation is part of a
> multigw configuration, and I have just updated one host in the secondary
> zone (all other hosts/zones are running 10.2.5).
>
> On the 10.2.7 server I cannot authenticate as a user (normally backed by
> OpenStack Keystone), but even worse I can also not authenticate with an
> admin user.
>
> Please see [1] for the results of performing a list bucket operation with
> python boto (script works against rgw 10.2.5)
>
> Also, if I try to authenticate from the 'master' rgw zone with a
> "radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
>
> "ERROR: failed to fetch datalog info"
>
> "failed to retrieve sync info: (13) Permission denied"
>
> The above errors correlates to the errors in the log on the server running
> 10.2.7 (debug level 20) at [2]
>
> I'm not sure what I have done wrong or can try next?
>
> By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
> authentication functionality
>
> [1]
> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
> <?xml version="1.0"
> encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000004-0058f8c86a-3fa2959-bbp-gva-secondary</RequestId><HostId>3fa2959-bbp-gva-secondary-bbp-gva</HostId></Error>
>
> [2]
> /bbpsrvc15.cscs.ch/admin/log
> 2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
> digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15
> auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
> 2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
> 2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
> 2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
> err_no=-2027 new_err_no=-2027
> 2017-04-20 16:43:04.916329 7ff87c6c0700 2 req 354:0.052585:s3:GET
> /admin/log:get_obj:op status=0
> 2017-04-20 16:43:04.916339 7ff87c6c0700 2 req 354:0.052595:s3:GET
> /admin/log:get_obj:http status=403
> 2017-04-20 16:43:04.916343 7ff87c6c0700 1 ====== req done
> req=0x7ff87c6ba710 op status=0 http_status=403 ======
> 2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
> 2017-04-20 16:43:04.916390 7ff87c6c0700 1 civetweb: 0x7ff990015610:
> 10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
> - -
> 2017-04-20 16:43:04.917212 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
> 2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1544: shard_id=20
> mdlog_marker=1_1492686039.901886_5551978.1
> sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
> 2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
> incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
> 2017-04-20 16:43:04.917236 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: init request
> 2017-04-20 16:43:04.917240 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status
> 2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
> blocked
> 2017-04-20 16:43:04.918285 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: reading shard status complete
> 2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
> marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
> 13:00:39.0.901886s
> 2017-04-20 16:43:04.918316 7ff9777e6700 20
> cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
> 2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
> shard_id=20: sending rest request
> 2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
> 20 14:43:04 2017
> 2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
> 14:43:04 2017
> 2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
> dest=/admin/log
> 2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET
>
> --
> Kind regards,
>
> Ben Morrice
>
> ______________________________________________________________________
> Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
> EPFL / BBP
> Biotech Campus
> Chemin des Mines 9
> 1202 Geneva
> Switzerland
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
admin
2017-04-26 11:18:42.929060 7f301d7f2700 20 RGWEnv::set(): HTTP_HOST: bbpsrvc15.cscs.ch
2017-04-26 11:18:42.929092 7f301d7f2700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2017-04-26 11:18:42.929095 7f301d7f2700 20 RGWEnv::set(): HTTP_DATE: Wed, 26 Apr 2017 09:18:42 GMT
2017-04-26 11:18:42.929097 7f301d7f2700 20 RGWEnv::set(): CONTENT_LENGTH: 0
2017-04-26 11:18:42.929099 7f301d7f2700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS A0QV52GF9GT3A13ADA6C:FReLIPp6EeM+BYZmFF7bfINOOC8=
2017-04-26 11:18:42.929102 7f301d7f2700 20 RGWEnv::set(): HTTP_USER_AGENT: Boto/2.46.1 Python/2.7.12 Linux/4.6.0-040600-generic
2017-04-26 11:18:42.929104 7f301d7f2700 20 RGWEnv::set(): REQUEST_METHOD: GET
2017-04-26 11:18:42.929106 7f301d7f2700 20 RGWEnv::set(): REQUEST_URI: /
2017-04-26 11:18:42.929107 7f301d7f2700 20 RGWEnv::set(): QUERY_STRING:
2017-04-26 11:18:42.929110 7f301d7f2700 20 RGWEnv::set(): REMOTE_USER:
2017-04-26 11:18:42.929111 7f301d7f2700 20 RGWEnv::set(): SCRIPT_URI: /
2017-04-26 11:18:42.929114 7f301d7f2700 20 RGWEnv::set(): SERVER_PORT: 80
2017-04-26 11:18:42.929115 7f301d7f2700 20 CONTENT_LENGTH=0
2017-04-26 11:18:42.929116 7f301d7f2700 20 HTTP_ACCEPT_ENCODING=identity
2017-04-26 11:18:42.929117 7f301d7f2700 20 HTTP_AUTHORIZATION=AWS A0QV52GF9GT3A13ADA6C:FReLIPp6EeM+BYZmFF7bfINOOC8=
2017-04-26 11:18:42.929118 7f301d7f2700 20 HTTP_DATE=Wed, 26 Apr 2017 09:18:42 GMT
2017-04-26 11:18:42.929118 7f301d7f2700 20 HTTP_HOST=bbpsrvc15.cscs.ch
2017-04-26 11:18:42.929119 7f301d7f2700 20 HTTP_USER_AGENT=Boto/2.46.1 Python/2.7.12 Linux/4.6.0-040600-generic
2017-04-26 11:18:42.929119 7f301d7f2700 20 QUERY_STRING=
2017-04-26 11:18:42.929120 7f301d7f2700 20 REMOTE_USER=
2017-04-26 11:18:42.929120 7f301d7f2700 20 REQUEST_METHOD=GET
2017-04-26 11:18:42.929121 7f301d7f2700 20 REQUEST_URI=/
2017-04-26 11:18:42.929121 7f301d7f2700 20 SCRIPT_URI=/
2017-04-26 11:18:42.929122 7f301d7f2700 20 SERVER_PORT=80
2017-04-26 11:18:42.929126 7f301d7f2700 1 ====== starting new request req=0x7f301d7ec710 =====
2017-04-26 11:18:42.929152 7f301d7f2700 2 req 1:0.000026::GET /::initializing for trans_id = tx000000000000000000001-00590065f2-400cf8b-bbp-gva-secondary
2017-04-26 11:18:42.929162 7f301d7f2700 10 rgw api priority: s3=5 s3website=4
2017-04-26 11:18:42.929165 7f301d7f2700 10 host=bbpsrvc15.cscs.ch
2017-04-26 11:18:42.929170 7f301d7f2700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-04-26 11:18:42.929174 7f301d7f2700 20 final domain/bucket subdomain=bbpsrvc15.cscs.ch domain= in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bbpsrvc15.cscs.ch/
2017-04-26 11:18:42.929216 7f301d7f2700 20 get_handler handler=25RGWHandler_REST_Bucket_S3
2017-04-26 11:18:42.929223 7f301d7f2700 10 handler=25RGWHandler_REST_Bucket_S3
2017-04-26 11:18:42.929225 7f301d7f2700 2 req 1:0.000099:s3:GET /::getting op 0
2017-04-26 11:18:42.929236 7f301d7f2700 10 op=25RGWListBucket_ObjStore_S3
2017-04-26 11:18:42.929247 7f301d7f2700 2 req 1:0.000112:s3:GET /:list_bucket:authorizing
2017-04-26 11:18:42.929265 7f301d7f2700 20 s3 keystone: trying keystone auth
2017-04-26 11:18:42.929290 7f301d7f2700 10 get_canon_resource(): dest=/bbpsrvc15.cscs.ch/
2017-04-26 11:18:42.929314 7f301d7f2700 20 found cached admin token
2017-04-26 11:18:42.929383 7f301d7f2700 20 sending request to https://bbpopenstack.epfl.ch:35357/v3/s3tokens
2017-04-26 11:18:43.037475 7f301d7f2700 0 Keystone token parse error: missing mandatory field access
2017-04-26 11:18:43.037496 7f301d7f2700 2 s3 keystone: token parsing failed
2017-04-26 11:18:43.037529 7f301d7f2700 20 get_system_obj_state: rctx=0x7f301d7eb1e0 obj=.bbp-gva-secondary.users:A0QV52GF9GT3A13ADA6C state=0x7f30d404e728 s->prefetch_data=0
2017-04-26 11:18:43.038449 7f301d7f2700 20 get_system_obj_state: s->obj_tag was set empty
2017-04-26 11:18:43.038466 7f301d7f2700 20 rados->read ofs=0 len=524288
2017-04-26 11:18:43.039137 7f301d7f2700 20 rados->read r=0 bl.length=17
2017-04-26 11:18:43.039169 7f301d7f2700 20 get_system_obj_state: rctx=0x7f301d7eaea0 obj=.bbp-gva-secondary.users.uid:admin-bbp-gva state=0x7f30d404f428 s->prefetch_data=0
2017-04-26 11:18:43.040082 7f301d7f2700 20 get_system_obj_state: s->obj_tag was set empty
2017-04-26 11:18:43.040106 7f301d7f2700 20 rados->read ofs=0 len=524288
2017-04-26 11:18:43.040871 7f301d7f2700 20 rados->read r=0 bl.length=341
2017-04-26 11:18:43.040925 7f301d7f2700 10 get_canon_resource(): dest=/bbpsrvc15.cscs.ch/
2017-04-26 11:18:43.040928 7f301d7f2700 10 auth_hdr:
GET
Wed, 26 Apr 2017 09:18:42 GMT
/bbpsrvc15.cscs.ch/
2017-04-26 11:18:43.040975 7f301d7f2700 15 calculated digest=p2oFMutvoZFu7UUaXCd7QbpDOlo=
2017-04-26 11:18:43.040976 7f301d7f2700 15 auth_sign=FReLIPp6EeM+BYZmFF7bfINOOC8=
2017-04-26 11:18:43.040979 7f301d7f2700 15 compare=-42
2017-04-26 11:18:43.040982 7f301d7f2700 10 failed to authorize request
2017-04-26 11:18:43.040984 7f301d7f2700 20 handler->ERRORHANDLER: err_no=-2027 new_err_no=-2027
2017-04-26 11:18:43.041065 7f301d7f2700 2 req 1:0.111939:s3:GET /:list_bucket:op status=0
2017-04-26 11:18:43.041069 7f301d7f2700 2 req 1:0.111943:s3:GET /:list_bucket:http status=403
2017-04-26 11:18:43.041075 7f301d7f2700 1 ====== req done req=0x7f301d7ec710 op status=0 http_status=403 ======
2017-04-26 11:18:43.041087 7f301d7f2700 20 process_request() returned -2027
2017-04-26 11:18:43.041125 7f301d7f2700 1 civetweb: 0x7f30d4004110: 128.178.97.85 - - [26/Apr/2017:11:18:42 +0200] "GET / HTTP/1.1" 403 0 - Boto/2.46.1 Python/2.7.12 Linux/4.6.0-040600-generic
2017-04-26 11:18:47.309637 7f312e7fc700 2 RGWDataChangesLog::ChangesRenewThread: start
2017-04-26 11:19:48.763484 7f30237fe700 20 RGWEnv::set(): HTTP_HOST: bbpsrvc15.cscs.ch
2017-04-26 11:19:48.763516 7f30237fe700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2017-04-26 11:19:48.763530 7f30237fe700 20 RGWEnv::set(): HTTP_DATE: Wed, 26 Apr 2017 09:19:48 GMT
2017-04-26 11:19:48.763532 7f30237fe700 20 RGWEnv::set(): CONTENT_LENGTH: 0
2017-04-26 11:19:48.763536 7f30237fe700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS a958d8c7f55c499aa3cc29b011bcb631:sxjBnGhLakp1qJwwPSdC/zGPKIQ=
2017-04-26 11:19:48.763541 7f30237fe700 20 RGWEnv::set(): HTTP_USER_AGENT: Boto/2.46.1 Python/2.7.12 Linux/4.6.0-040600-generic
2017-04-26 11:19:48.763544 7f30237fe700 20 RGWEnv::set(): REQUEST_METHOD: GET
2017-04-26 11:19:48.763546 7f30237fe700 20 RGWEnv::set(): REQUEST_URI: /
2017-04-26 11:19:48.763549 7f30237fe700 20 RGWEnv::set(): QUERY_STRING:
2017-04-26 11:19:48.763551 7f30237fe700 20 RGWEnv::set(): REMOTE_USER:
2017-04-26 11:19:48.763553 7f30237fe700 20 RGWEnv::set(): SCRIPT_URI: /
2017-04-26 11:19:48.763558 7f30237fe700 20 RGWEnv::set(): SERVER_PORT: 80
2017-04-26 11:19:48.763560 7f30237fe700 20 CONTENT_LENGTH=0
2017-04-26 11:19:48.763562 7f30237fe700 20 HTTP_ACCEPT_ENCODING=identity
2017-04-26 11:19:48.763563 7f30237fe700 20 HTTP_AUTHORIZATION=AWS a958d8c7f55c499aa3cc29b011bcb631:sxjBnGhLakp1qJwwPSdC/zGPKIQ=
2017-04-26 11:19:48.763565 7f30237fe700 20 HTTP_DATE=Wed, 26 Apr 2017 09:19:48 GMT
2017-04-26 11:19:48.763567 7f30237fe700 20 HTTP_HOST=bbpsrvc15.cscs.ch
2017-04-26 11:19:48.763568 7f30237fe700 20 HTTP_USER_AGENT=Boto/2.46.1 Python/2.7.12 Linux/4.6.0-040600-generic
2017-04-26 11:19:48.763569 7f30237fe700 20 QUERY_STRING=
2017-04-26 11:19:48.763570 7f30237fe700 20 REMOTE_USER=
2017-04-26 11:19:48.763571 7f30237fe700 20 REQUEST_METHOD=GET
2017-04-26 11:19:48.763572 7f30237fe700 20 REQUEST_URI=/
2017-04-26 11:19:48.763573 7f30237fe700 20 SCRIPT_URI=/
2017-04-26 11:19:48.763574 7f30237fe700 20 SERVER_PORT=80
2017-04-26 11:19:48.763592 7f30237fe700 1 ====== starting new request req=0x7f30237f8710 =====
2017-04-26 11:19:48.763649 7f30237fe700 2 req 2:0.000063::GET /::initializing for trans_id = tx000000000000000000002-0059006634-400cf8b-bbp-gva-secondary
2017-04-26 11:19:48.763664 7f30237fe700 10 rgw api priority: s3=5 s3website=4
2017-04-26 11:19:48.763667 7f30237fe700 10 host=bbpsrvc15.cscs.ch
2017-04-26 11:19:48.763672 7f30237fe700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-04-26 11:19:48.763686 7f30237fe700 20 final domain/bucket subdomain=bbpsrvc15.cscs.ch domain= in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/bbpsrvc15.cscs.ch/
2017-04-26 11:19:48.763782 7f30237fe700 20 get_handler handler=25RGWHandler_REST_Bucket_S3
2017-04-26 11:19:48.763795 7f30237fe700 10 handler=25RGWHandler_REST_Bucket_S3
2017-04-26 11:19:48.763800 7f30237fe700 2 req 2:0.000223:s3:GET /::getting op 0
2017-04-26 11:19:48.763834 7f30237fe700 10 op=25RGWListBucket_ObjStore_S3
2017-04-26 11:19:48.763837 7f30237fe700 2 req 2:0.000261:s3:GET /:list_bucket:authorizing
2017-04-26 11:19:48.763844 7f30237fe700 20 s3 keystone: trying keystone auth
2017-04-26 11:19:48.763877 7f30237fe700 10 get_canon_resource(): dest=/bbpsrvc15.cscs.ch/
2017-04-26 11:19:48.763922 7f30237fe700 20 found cached admin token
2017-04-26 11:19:48.764023 7f30237fe700 20 sending request to https://bbpopenstack.epfl.ch:35357/v3/s3tokens
2017-04-26 11:19:48.829058 7f30237fe700 20 get_system_obj_state: rctx=0x7f30237f71e0 obj=.bbp-gva-secondary.users:a958d8c7f55c499aa3cc29b011bcb631 state=0x7f30ec04f528 s->prefetch_data=0
2017-04-26 11:19:48.830191 7f30237fe700 5 error reading user info, uid=a958d8c7f55c499aa3cc29b011bcb631 can't authenticate
2017-04-26 11:19:48.830208 7f30237fe700 10 failed to authorize request
2017-04-26 11:19:48.830212 7f30237fe700 20 handler->ERRORHANDLER: err_no=-2027 new_err_no=-2027
2017-04-26 11:19:48.830305 7f30237fe700 2 req 2:0.066728:s3:GET /:list_bucket:op status=0
2017-04-26 11:19:48.830320 7f30237fe700 2 req 2:0.066744:s3:GET /:list_bucket:http status=403
2017-04-26 11:19:48.830328 7f30237fe700 1 ====== req done req=0x7f30237f8710 op status=0 http_status=403 ======
2017-04-26 11:19:48.830348 7f30237fe700 20 process_request() returned -2027
2017-04-26 11:19:48.830387 7f30237fe700 1 civetweb: 0x7f30ec004110: 128.178.97.85 - - [26/Apr/2017:11:19:48 +0200] "GET / HTTP/1.1" 403 0 - Boto/2.46.1 Python/2.7.12 Linux/4.6.0-040600-generic
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
