Re: RGW 10.2.5->10.2.7 authentication fail?

Ben Morrice <ben.morrice@xxxxxxx> · Fri, 28 Apr 2017 10:06:41 +0200

Hello again,

I can work around this issue. If the host header is an IP address, the 
request is treated as a virtual:

So if I auth to to my backends via IP, things work as expected.

Kind regards,

Ben Morrice

______________________________________________________________________
Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 28/04/17 09:26, Ben Morrice wrote:
Hello Radek,

Thanks again for your anaylsis.

I can confirm on 10.2.7, if I remove the conf "rgw dns name" I can 
auth to directly to the radosgw host.

In our environment we terminate SSL and route connections via haproxy, 
but it's still sometimes useful to be able to communicate directly to 
the backend radosgw server.

It seems that it's not possible to set multiple "rgw dns name" entries 
in ceph.conf

Is the only solution to modify the zonegroup and populate the 
'hostnames' array with all backend server hostnames as well as the 
hostname terminated by haproxy?

Kind regards,

Ben Morrice

______________________________________________________________________
Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 27/04/17 13:53, Radoslaw Zarzynski wrote:
Bingo! From the 10.2.5-admin:

   GET

   Thu, 27 Apr 2017 07:49:59 GMT
   /

And also:

   2017-04-27 09:49:59.117447 7f4a90ff9700 20 subdomain= domain=
in_hosted_domain=0 in_hosted_domain_s3website=0
   2017-04-27 09:49:59.117449 7f4a90ff9700 20 final domain/bucket
subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
s->info.domain= s->info.request_uri=/

The most interesting part is the "final ... in_hosted_domain=0".
It looks we need to dig around RGWREST::preprocess(),
rgw_find_host_in_domains() & company.

There is a commit introduced in v10.2.6 that touches this area [1].
I'm definitely not saying it's the root cause. It might be that a change
in the code just unhidden a configuration issue [2].

I will talk about the problem on the today's sync-up.

Thanks for the logs!
Regards,
Radek

[1] 
https://github.com/ceph/ceph/commit/c9445faf7fac2ccb8a05b53152c0ca16d7f4c6d0
[2] http://tracker.ceph.com/issues/17440

On Thu, Apr 27, 2017 at 10:11 AM, Ben Morrice <ben.morrice@xxxxxxx> 
wrote:
Hello Radek,

Thank-you for your analysis so far! Please find attached logs for 
both the
admin user and a keystone backed user from 10.2.5 (same host as 
before, I
have simply downgraded the packages). Both users can authenticate 
and list
buckets on 10.2.5.

Also - I tried version 10.2.6 and see the same behavior as 10.2.7, 
so the
bug i'm hitting looks like it was introduced in 10.2.6

Kind regards,

Ben Morrice

______________________________________________________________________
Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

On 27/04/17 04:45, Radoslaw Zarzynski wrote:
Thanks for the logs, Ben.

It looks that two completely different authenticators have failed:
the local, RADOS-backed auth (admin.txt) and Keystone-based
one as well. In the second case I'm pretty sure that Keystone has
rejected [1][2] to authenticate provided signature/StringToSign.
RGW tried to fallback to the local auth which obviously didn't have
any chance as the credentials were stored remotely. This explains
the presence of "error reading user info" in the user-keystone.txt.

What is common for both scenarios are the low-level things related
to StringToSign crafting/signature generation at RadosGW's side.
Following one has been composed for the request from admin.txt:

    GET

    Wed, 26 Apr 2017 09:18:42 GMT
    /bbpsrvc15.cscs.ch/

If you could provide a similar log from v10.2.5, I would be really
grateful.

Regards,
Radek

[1]
https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_rest_s3.cc#L3269-L3272 

[2] 
https://github.com/ceph/ceph/blob/v10.2.7/src/rgw/rgw_common.h#L170

On Wed, Apr 26, 2017 at 11:29 AM, Morrice Ben <ben.morrice@xxxxxxx> 
wrote:
Hello Radek,

Please find attached the failed request for both the admin user and a
standard user (backed by keystone).

Kind regards,

Ben Morrice

______________________________________________________________________ 

Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

________________________________________
From: Radoslaw Zarzynski <rzarzynski@xxxxxxxxxxxx>
Sent: Tuesday, April 25, 2017 7:38 PM
To: Morrice Ben
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re:  RGW 10.2.5->10.2.7 authentication fail?

Hello Ben,

Could you provide full RadosGW's log for the failed request?
I mean the lines starting from header listing, through the start
marker ("====== starting new request...") till the end marker?

At the moment we can't see any details related to the signature
calculation.

Regards,
Radek

On Thu, Apr 20, 2017 at 5:08 PM, Ben Morrice <ben.morrice@xxxxxxx> 
wrote:
Hi all,

I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7
(RHEL7)
and authentication is in a very bad state. This installation is 
part of
a
multigw configuration, and I have just updated one host in the 
secondary
zone (all other hosts/zones are running 10.2.5).

On the 10.2.7 server I cannot authenticate as a user (normally 
backed by
OpenStack Keystone), but even worse I can also not authenticate 
with an
admin user.

Please see [1] for the results of performing a list bucket operation
with
python boto (script works against rgw 10.2.5)

Also, if I try to authenticate from the 'master' rgw zone with a
"radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:

"ERROR: failed to fetch datalog info"

"failed to retrieve sync info: (13) Permission denied"

The above errors correlates to the errors in the log on the server
running
10.2.7 (debug level 20) at [2]

I'm not sure what I have done wrong or can try next?

By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
authentication functionality

[1]
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0"

encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000004-0058f8c86a-3fa2959-bbp-gva-secondary</RequestId><HostId>3fa2959-bbp-gva-secondary-bbp-gva</HostId></Error> 

[2]
/bbpsrvc15.cscs.ch/admin/log
2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
2017-04-20 16:43:04.916255 7ff87c6c0700 15
auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize 
request
2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
err_no=-2027 new_err_no=-2027
2017-04-20 16:43:04.916329 7ff87c6c0700  2 req 354:0.052585:s3:GET
/admin/log:get_obj:op status=0
2017-04-20 16:43:04.916339 7ff87c6c0700  2 req 354:0.052595:s3:GET
/admin/log:get_obj:http status=403
2017-04-20 16:43:04.916343 7ff87c6c0700  1 ====== req done
req=0x7ff87c6ba710 op status=0 http_status=403 ======
2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() 
returned
-2027
2017-04-20 16:43:04.916390 7ff87c6c0700  1 civetweb: 0x7ff990015610:
10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log 
HTTP/1.1"
403 0
- -
2017-04-20 16:43:04.917212 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: 
operate()
2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
incremental_sync:1544: shard_id=20
mdlog_marker=1_1492686039.901886_5551978.1
sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
2017-04-20 16:43:04.917236 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine:
operate()
2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: init request
2017-04-20 16:43:04.917240 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine:
operate()
2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status
2017-04-20 16:43:04.917303 7ff9777e6700 20 run: 
stack=0x7ff97000d420 is
io
blocked
2017-04-20 16:43:04.918285 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine:
operate()
2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status complete
2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: 
shard_id=20
marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
13:00:39.0.901886s
2017-04-20 16:43:04.918316 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine:
operate()
2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: sending rest request
2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): 
HTTP_DATE: Thu
Apr
20 14:43:04 2017
2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
14:43:04 2017
2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
dest=/admin/log
2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical 
header:
GET

--
Kind regards,

Ben Morrice

______________________________________________________________________ 

Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com