----- Original Message -----
> From: "Neville" <neville.taylor@xxxxxxxxxxxxx>
> To: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx>
> Cc: ceph-users@xxxxxxxxxxxxxx
> Sent: Monday, March 30, 2015 6:49:29 AM
> Subject: Re: Radosgw authorization failed
>
>
> > Date: Wed, 25 Mar 2015 11:43:44 -0400
> > From: yehuda@xxxxxxxxxx
> > To: neville.taylor@xxxxxxxxxxxxx
> > CC: ceph-users@xxxxxxxxxxxxxx
> > Subject: Re: Radosgw authorization failed
> >
> >
> >
> > ----- Original Message -----
> > > From: "Neville" <neville.taylor@xxxxxxxxxxxxx>
> > > To: ceph-users@xxxxxxxxxxxxxx
> > > Sent: Wednesday, March 25, 2015 8:16:39 AM
> > > Subject: Radosgw authorization failed
> > >
> > > Hi all,
> > >
> > > I'm testing backup product which supports Amazon S3 as target for Archive
> > > storage and I'm trying to setup a Ceph cluster configured with the S3 API
> > > to
> > > use as an internal target for backup archives instead of AWS.
> > >
> > > I've followed the online guide for setting up Radosgw and created a
> > > default
> > > region and zone based on the AWS naming convention US-East-1. I'm not
> > > sure
> > > if this is relevant but since I was having issues I thought it might need
> > > to
> > > be the same.
> > >
> > > I've tested the radosgw using boto.s3 and it seems to work ok i.e. I can
> > > create a bucket, create a folder, list buckets etc. The problem is when
> > > the
> > > backup software tries to create an object I get an authorization failure.
> > > It's using the same user/access/secret as I'm using from boto.s3 and I'm
> > > sure the creds are right as it lets me create the initial connection, it
> > > just fails when trying to create an object (backup folder).
> > >
> > > Here's the extract from the radosgw log:
> > >
> > > ---------------------------------------------------------------------------------------------------------------------------------------------
> > > 2015-03-25 15:07:26.449227 7f1050dc7700 2 req 5:0.000419:s3:GET
> > > /:list_bucket:init op
> > > 2015-03-25 15:07:26.449232 7f1050dc7700 2 req 5:0.000424:s3:GET
> > > /:list_bucket:verifying op mask
> > > 2015-03-25 15:07:26.449234 7f1050dc7700 20 required_mask= 1
> > > user.op_mask=7
> > > 2015-03-25 15:07:26.449235 7f1050dc7700 2 req 5:0.000427:s3:GET
> > > /:list_bucket:verifying op permissions
> > > 2015-03-25 15:07:26.449237 7f1050dc7700 5 Searching permissions for
> > > uid=test
> > > mask=49
> > > 2015-03-25 15:07:26.449238 7f1050dc7700 5 Found permission: 15
> > > 2015-03-25 15:07:26.449239 7f1050dc7700 5 Searching permissions for
> > > group=1
> > > mask=49
> > > 2015-03-25 15:07:26.449240 7f1050dc7700 5 Found permission: 15
> > > 2015-03-25 15:07:26.449241 7f1050dc7700 5 Searching permissions for
> > > group=2
> > > mask=49
> > > 2015-03-25 15:07:26.449242 7f1050dc7700 5 Found permission: 15
> > > 2015-03-25 15:07:26.449243 7f1050dc7700 5 Getting permissions id=test
> > > owner=test perm=1
> > > 2015-03-25 15:07:26.449244 7f1050dc7700 10 uid=test requested perm
> > > (type)=1,
> > > policy perm=1, user_perm_mask=1, acl perm=1
> > > 2015-03-25 15:07:26.449245 7f1050dc7700 2 req 5:0.000437:s3:GET
> > > /:list_bucket:verifying op params
> > > 2015-03-25 15:07:26.449247 7f1050dc7700 2 req 5:0.000439:s3:GET
> > > /:list_bucket:executing
> > > 2015-03-25 15:07:26.449252 7f1050dc7700 10 cls_bucket_list
> > > test1(@{i=.us-east.rgw.buckets.index}.us-east.rgw.buckets[us-east.280959.2])
> > > start num 1001
> > > 2015-03-25 15:07:26.450828 7f1050dc7700 2 req 5:0.002020:s3:GET
> > > /:list_bucket:http status=200
> > > 2015-03-25 15:07:26.450832 7f1050dc7700 1 ====== req done
> > > req=0x7f107000e2e0
> > > http_status=200 ======
> > > 2015-03-25 15:07:26.516999 7f1069df9700 20 enqueued request
> > > req=0x7f107000f0e0
> > > 2015-03-25 15:07:26.517006 7f1069df9700 20 RGWWQ:
> > > 2015-03-25 15:07:26.517007 7f1069df9700 20 req: 0x7f107000f0e0
> > > 2015-03-25 15:07:26.517010 7f1069df9700 10 allocated request
> > > req=0x7f107000f6b0
> > > 2015-03-25 15:07:26.517021 7f1058dd7700 20 dequeued request
> > > req=0x7f107000f0e0
> > > 2015-03-25 15:07:26.517023 7f1058dd7700 20 RGWWQ: empty
> > > 2015-03-25 15:07:26.517081 7f1058dd7700 20 CONTENT_LENGTH=88
> > > 2015-03-25 15:07:26.517084 7f1058dd7700 20
> > > CONTENT_TYPE=application/octet-stream
> > > 2015-03-25 15:07:26.517085 7f1058dd7700 20 CONTEXT_DOCUMENT_ROOT=/var/www
> > > 2015-03-25 15:07:26.517086 7f1058dd7700 20 CONTEXT_PREFIX=
> > > 2015-03-25 15:07:26.517087 7f1058dd7700 20 DOCUMENT_ROOT=/var/www
> > > 2015-03-25 15:07:26.517088 7f1058dd7700 20 FCGI_ROLE=RESPONDER
> > > 2015-03-25 15:07:26.517089 7f1058dd7700 20 GATEWAY_INTERFACE=CGI/1.1
> > > 2015-03-25 15:07:26.517090 7f1058dd7700 20 HTTP_AUTHORIZATION=AWS
> > > F79L68W19B3GCLOSE3F8:AcXqtvlBzBMpwdL+WuhDRoLT/Bs=
> > > 2015-03-25 15:07:26.517091 7f1058dd7700 20 HTTP_CONNECTION=Keep-Alive
> > > 2015-03-25 15:07:26.517092 7f1058dd7700 20 HTTP_DATE=Wed, 25 Mar 2015
> > > 15:07:26 GMT
> > > 2015-03-25 15:07:26.517092 7f1058dd7700 20 HTTP_EXPECT=100-continue
> > > 2015-03-25 15:07:26.517093 7f1058dd7700 20
> > > HTTP_HOST=test1.devops-os-cog01.devops.local
> > > 2015-03-25 15:07:26.517094 7f1058dd7700 20
> > > HTTP_USER_AGENT=aws-sdk-java/unknown-version Windows_Server_2008_R2/6.1
> > > Java_HotSpot(TM)_Client_VM/24.55-b03
> > > 2015-03-25 15:07:26.517096 7f1058dd7700 20
> > > HTTP_X_AMZ_META_CREATIONTIME=2015-03-25T15:07:26
> > > 2015-03-25 15:07:26.517097 7f1058dd7700 20 HTTP_X_AMZ_META_SIZE=88
> > > 2015-03-25 15:07:26.517098 7f1058dd7700 20
> > > HTTP_X_AMZ_STORAGE_CLASS=STANDARD
> > > 2015-03-25 15:07:26.517099 7f1058dd7700 20 HTTPS=on
> > > 2015-03-25 15:07:26.517100 7f1058dd7700 20
> > > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> > > 2015-03-25 15:07:26.517100 7f1058dd7700 20 QUERY_STRING=
> > > 2015-03-25 15:07:26.517101 7f1058dd7700 20 REMOTE_ADDR=10.40.41.106
> > > 2015-03-25 15:07:26.517102 7f1058dd7700 20 REMOTE_PORT=55439
> > > 2015-03-25 15:07:26.517103 7f1058dd7700 20 REQUEST_METHOD=PUT
> > > 2015-03-25 15:07:26.517104 7f1058dd7700 20 REQUEST_SCHEME=https
> > > 2015-03-25 15:07:26.517105 7f1058dd7700 20
> > > REQUEST_URI=/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> > > 2015-03-25 15:07:26.517106 7f1058dd7700 20
> > > SCRIPT_FILENAME=/var/www/s3gw.fcgi
> > > 2015-03-25 15:07:26.517107 7f1058dd7700 20
> > > SCRIPT_NAME=/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> > > 2015-03-25 15:07:26.517108 7f1058dd7700 20
> > > SCRIPT_URI=https://test1.devops-os-cog01.devops.local/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> > > 2015-03-25 15:07:26.517109 7f1058dd7700 20
> > > SCRIPT_URL=/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> > > 2015-03-25 15:07:26.517110 7f1058dd7700 20 SERVER_ADDR=10.40.41.64
> > > 2015-03-25 15:07:26.517111 7f1058dd7700 20
> > > SERVER_ADMIN=no-one@devops.local
> > > 2015-03-25 15:07:26.517112 7f1058dd7700 20
> > > SERVER_NAME=test1.devops-os-cog01.devops.local
> > > 2015-03-25 15:07:26.517113 7f1058dd7700 20 SERVER_PORT=443
> > > 2015-03-25 15:07:26.517114 7f1058dd7700 20 SERVER_PORT_SECURE=443
> > > 2015-03-25 15:07:26.517115 7f1058dd7700 20 SERVER_PROTOCOL=HTTP/1.1
> > > 2015-03-25 15:07:26.517116 7f1058dd7700 20 SERVER_SIGNATURE=
> > > 2015-03-25 15:07:26.517117 7f1058dd7700 20 SERVER_SOFTWARE=Apache/2.4.7
> > > (Ubuntu)
> > > 2015-03-25 15:07:26.517119 7f1058dd7700 1 ====== starting new request
> > > req=0x7f107000f0e0 =====
> > > 2015-03-25 15:07:26.517129 7f1058dd7700 2 req 6:0.000010::PUT
> > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3::initializing
> > > 2015-03-25 15:07:26.517132 7f1058dd7700 10
> > > host=test1.devops-os-cog01.devops.local
> > > rgw_dns_name=devops-os-cog01.devops.local
> > > 2015-03-25 15:07:26.517143 7f1058dd7700 10 meta>>
> > > HTTP_X_AMZ_META_CREATIONTIME
> > > 2015-03-25 15:07:26.517147 7f1058dd7700 10 meta>> HTTP_X_AMZ_META_SIZE
> > > 2015-03-25 15:07:26.517150 7f1058dd7700 10 meta>>
> > > HTTP_X_AMZ_STORAGE_CLASS
> > > 2015-03-25 15:07:26.517155 7f1058dd7700 10 x>>
> > > x-amz-meta-creationtime:2015-03-25T15:07:26
> > > 2015-03-25 15:07:26.517156 7f1058dd7700 10 x>> x-amz-meta-size:88
> > > 2015-03-25 15:07:26.517157 7f1058dd7700 10 x>>
> > > x-amz-storage-class:STANDARD
> > > 2015-03-25 15:07:26.517167 7f1058dd7700 10
> > > s->object=ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 s->bucket=test1
> > > 2015-03-25 15:07:26.517171 7f1058dd7700 2 req 6:0.000052:s3:PUT
> > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3::getting op
> > > 2015-03-25 15:07:26.517175 7f1058dd7700 2 req 6:0.000055:s3:PUT
> > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3:put_obj:authorizing
> > > 2015-03-25 15:07:26.517209 7f1058dd7700 20 get_obj_state:
> > > rctx=0x7f106c025ce0
> > > obj=.us-east.users:F79L68W19B3GCLOSE3F8 state=0x7f106c0260b8
> > > s->prefetch_data=0
> > > 2015-03-25 15:07:26.517215 7f1058dd7700 10 cache get:
> > > name=.us-east.users+F79L68W19B3GCLOSE3F8 : hit
> > > 2015-03-25 15:07:26.517219 7f1058dd7700 20 get_obj_state: s->obj_tag was
> > > set
> > > empty
> > > 2015-03-25 15:07:26.517224 7f1058dd7700 10 cache get:
> > > name=.us-east.users+F79L68W19B3GCLOSE3F8 : hit
> > > 2015-03-25 15:07:26.517248 7f1058dd7700 20 get_obj_state:
> > > rctx=0x7f106c025ce0
> > > obj=.us-east.users.uid:test state=0x7f106c0267c8 s->prefetch_data=0
> > > 2015-03-25 15:07:26.517252 7f1058dd7700 10 cache get:
> > > name=.us-east.users.uid+test : hit
> > > 2015-03-25 15:07:26.517256 7f1058dd7700 20 get_obj_state: s->obj_tag was
> > > set
> > > empty
> > > 2015-03-25 15:07:26.517261 7f1058dd7700 10 cache get:
> > > name=.us-east.users.uid+test : hit
> > > 2015-03-25 15:07:26.517296 7f1058dd7700 10 get_canon_resource():
> > > dest=/test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> > > 2015-03-25 15:07:26.517298 7f1058dd7700 10 auth_hdr:
> > > PUT
> > > application/octet-stream
> > > Wed, 25 Mar 2015 15:07:26 GMT
> > > x-amz-meta-creationtime:2015-03-25T15:07:26
> > > x-amz-meta-size:88
> > > x-amz-storage-class:STANDARD
> > > /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> > > 2015-03-25 15:07:26.517341 7f1058dd7700 15 calculated
> > > digest=coYY2IBRECc42IXE7HQlYxBcqfU=
> > > 2015-03-25 15:07:26.517343 7f1058dd7700 15
> > > auth_sign=AcXqtvlBzBMpwdL+WuhDRoLT/Bs=
> > > 2015-03-25 15:07:26.517344 7f1058dd7700 15 compare=-34
> > > 2015-03-25 15:07:26.517346 7f1058dd7700 10 failed to authorize request
> > > 2015-03-25 15:07:26.517363 7f1058dd7700 2 req 6:0.000244:s3:PUT
> > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3:put_obj:http status=403
> > > 2015-03-25 15:07:26.517367 7f1058dd7700 1 ====== req done
> > > req=0x7f107000f0e0
> > > http_status=403 ======
> > > 2015-03-25 15:07:26.517374 7f1058dd7700 20 process_request() returned -1
> > > 2015-03-25 15:07:28.058030 7f1088ff9700 2
> > > RGWDataChangesLog::ChangesRenewThread: start
> >
> > Nothing pops up as being wrong. Assuming the first and second requests were
> > done using the same credentials, I would guess that it might be the client
> > incorrectly creating the canonical representation of the request. Maybe it
> > doesn't use the x-amz-storage-class correctly there?. Can you try
> > verifying what is the plain text signature string that it's using? Also,
> > try disabling the storage class param, see if it makes any difference.
> >
> > Yehuda
> I've looked through the application logs but it doesn't appear to capture
> what the request string is that it's using. I also can't find out any way of
> disabling the x-amz-storage-class unfortunately. I did test the application
> against AWS S3 and the functionality I'm trying to use seems to work fine so
> it suggests it is a problem with my Ceph implementation. To help
> troubleshoot I've tried to capture some different requests for comparison:
> GET request from backup application (Successful):
> 2015-03-30 11:55:05.331145 7f103f5a4700 10 get_canon_resource(): dest=/test1/
> 2015-03-30 11:55:05.331148 7f103f5a4700 10 auth_hdr:
> GET
> application/x-www-form-urlencoded; charset=utf-8
> Mon, 30 Mar 2015 10:55:05 GMT
> /test1/
> 2015-03-30 11:55:05.331195 7f103f5a4700 15 calculated
> digest=4tEUx7ERu+joNwYWrQboPhy/+Y4=
> 2015-03-30 11:55:05.331197 7f103f5a4700 15
> auth_sign=4tEUx7ERu+joNwYWrQboPhy/+Y4=
> 2015-03-30 11:55:05.331198 7f103f5a4700 15 compare=0
> 2015-03-30 11:55:05.331200 7f103f5a4700 2 req 167:0.000252:s3:GET
> /:list_bucket:reading permissions
> 2015-03-30 11:55:05.331239 7f103f5a4700 20 get_obj_state: rctx=0x7f103f5a3230
> obj=.us-east.domain.rgw:test1 state=0x7f109d479098 s->prefetch_data=0
> 2015-03-30 11:55:05.331245 7f103f5a4700 10 cache get:
> name=.us-east.domain.rgw+test1 : hit
> PUT request from backup application (Fails):
> 2015-03-30 11:55:05.414753 7f105cddf700 10 get_canon_resource():
> dest=/test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> 2015-03-30 11:55:05.414755 7f105cddf700 10 auth_hdr:
> PUT
> application/octet-stream
> Mon, 30 Mar 2015 10:55:05 GMT
> x-amz-meta-creationtime:2015-03-30T10:55:05
> x-amz-meta-size:88
> x-amz-storage-class:STANDARD
> /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3
> 2015-03-30 11:55:05.414800 7f105cddf700 15 calculated
> digest=DtPnePmBb00mfJgAr/z+uW70Unc=
> 2015-03-30 11:55:05.414802 7f105cddf700 15
> auth_sign=iK1Wkt0uoIif9udGTf1P1fqF6Bk=
> 2015-03-30 11:55:05.414803 7f105cddf700 15 compare=37
> 2015-03-30 11:55:05.414805 7f105cddf700 10 failed to authorize request
> PUT request from "S3 Browser" application using S3 compatible connection
> (Successful):
> 2015-03-30 13:07:47.433691 7f10445ae700 10 get_canon_resource():
> dest=/test1/february/?acl
> 2015-03-30 13:07:47.433694 7f10445ae700 10 auth_hdr:
> PUT
> x-amz-date:Mon, 30 Mar 2015 12:07:47 GMT
> /test1/february/?acl
> 2015-03-30 13:07:47.433740 7f10445ae700 15 calculated
> digest=9g91r6RZ3YCZawt+H4YuGaSlffQ=
> 2015-03-30 13:07:47.433742 7f10445ae700 15
> auth_sign=9g91r6RZ3YCZawt+H4YuGaSlffQ=
> 2015-03-30 13:07:47.433743 7f10445ae700 15 compare=0
> 2015-03-30 13:07:47.433745 7f10445ae700 2 req 183:0.000227:s3:PUT
> /test1/february/:put_acls:reading permissions
> PUT request using boto S3 (Successful):
> 2015-03-30 13:26:29.394718 7f103f5a4700 10 get_canon_resource():
> dest=/test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3/
> 2015-03-30 13:26:29.394721 7f103f5a4700 10 auth_hdr:
> PUT
> 1B2M2Y8AsgTpgAmY7PhCfg==
> application/octet-stream
> Mon, 30 Mar 2015 12:26:02 GMT
> /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3/
> 2015-03-30 13:26:29.394821 7f103f5a4700 15 calculated
> digest=ljvCTLAqySSfcJ3baWfK5776kJM=
> 2015-03-30 13:26:29.394823 7f103f5a4700 15
> auth_sign=ljvCTLAqySSfcJ3baWfK5776kJM=
> 2015-03-30 13:26:29.394824 7f103f5a4700 15 compare=0
> 2015-03-30 13:26:29.394828 7f103f5a4700 2 req 188:0.000414:s3:PUT
> /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3/:put_obj:reading
> permissions
> Does this help at all? Is it fair to assume it's not the application if it
> works ok with AWS S3? I'm wondering if there is a bug in the S3
> implementation on Ceph which mean it handles one of these headers
> incorrectly:
> x-amz-meta-creationtime
> x-amz-meta-size
> x-amz-storage-class
The x-amz-meta-* headers are standard user defined headers that we support and test regularly. As for the storage-class header, we still accept it as a valid input, and we sign it as required (just verified it again).
With regard to it working against S3, but not against rgw, it is possible that the library is using a different method of authentication against the former (AWS4 vs AWS). It's hard for me to say whether the culprit is the storage-class, it could be something else that we don't see in these logs. In order to be sure we'll need to compare the plain text canonicalized resource string that the library signs. Maybe you could modify its source code and add a debug printing where needed?
Yehuda
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
