|
> Date: Wed, 25 Mar 2015 11:43:44 -0400 > From: yehuda@xxxxxxxxxx > To: neville.taylor@xxxxxxxxxxxxx > CC: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Radosgw authorization failed > > > > ----- Original Message ----- > > From: "Neville" <neville.taylor@xxxxxxxxxxxxx> > > To: ceph-users@xxxxxxxxxxxxxx > > Sent: Wednesday, March 25, 2015 8:16:39 AM > > Subject: Radosgw authorization failed > > > > Hi all, > > > > I'm testing backup product which supports Amazon S3 as target for Archive > > storage and I'm trying to setup a Ceph cluster configured with the S3 API to > > use as an internal target for backup archives instead of AWS. > > > > I've followed the online guide for setting up Radosgw and created a default > > region and zone based on the AWS naming convention US-East-1. I'm not sure > > if this is relevant but since I was having issues I thought it might need to > > be the same. > > > > I've tested the radosgw using boto.s3 and it seems to work ok i.e. I can > > create a bucket, create a folder, list buckets etc. The problem is when the > > backup software tries to create an object I get an authorization failure. > > It's using the same user/access/secret as I'm using from boto.s3 and I'm > > sure the creds are right as it lets me create the initial connection, it > > just fails when trying to create an object (backup folder). > > > > Here's the extract from the radosgw log: > > > > --------------------------------------------------------------------------------------------------------------------------------------------- > > 2015-03-25 15:07:26.449227 7f1050dc7700 2 req 5:0.000419:s3:GET > > /:list_bucket:init op > > 2015-03-25 15:07:26.449232 7f1050dc7700 2 req 5:0.000424:s3:GET > > /:list_bucket:verifying op mask > > 2015-03-25 15:07:26.449234 7f1050dc7700 20 required_mask= 1 user.op_mask=7 > > 2015-03-25 15:07:26.449235 7f1050dc7700 2 req 5:0.000427:s3:GET > > /:list_bucket:verifying op permissions > > 2015-03-25 15:07:26.449237 7f1050dc7700 5 Searching permissions for uid=test > > mask=49 > > 2015-03-25 15:07:26.449238 7f1050dc7700 5 Found permission: 15 > > 2015-03-25 15:07:26.449239 7f1050dc7700 5 Searching permissions for group=1 > > mask=49 > > 2015-03-25 15:07:26.449240 7f1050dc7700 5 Found permission: 15 > > 2015-03-25 15:07:26.449241 7f1050dc7700 5 Searching permissions for group=2 > > mask=49 > > 2015-03-25 15:07:26.449242 7f1050dc7700 5 Found permission: 15 > > 2015-03-25 15:07:26.449243 7f1050dc7700 5 Getting permissions id=test > > owner=test perm=1 > > 2015-03-25 15:07:26.449244 7f1050dc7700 10 uid=test requested perm (type)=1, > > policy perm=1, user_perm_mask=1, acl perm=1 > > 2015-03-25 15:07:26.449245 7f1050dc7700 2 req 5:0.000437:s3:GET > > /:list_bucket:verifying op params > > 2015-03-25 15:07:26.449247 7f1050dc7700 2 req 5:0.000439:s3:GET > > /:list_bucket:executing > > 2015-03-25 15:07:26.449252 7f1050dc7700 10 cls_bucket_list > > test1(@{i=.us-east.rgw.buckets.index}.us-east.rgw.buckets[us-east.280959.2]) > > start num 1001 > > 2015-03-25 15:07:26.450828 7f1050dc7700 2 req 5:0.002020:s3:GET > > /:list_bucket:http status=200 > > 2015-03-25 15:07:26.450832 7f1050dc7700 1 ====== req done req=0x7f107000e2e0 > > http_status=200 ====== > > 2015-03-25 15:07:26.516999 7f1069df9700 20 enqueued request > > req=0x7f107000f0e0 > > 2015-03-25 15:07:26.517006 7f1069df9700 20 RGWWQ: > > 2015-03-25 15:07:26.517007 7f1069df9700 20 req: 0x7f107000f0e0 > > 2015-03-25 15:07:26.517010 7f1069df9700 10 allocated request > > req=0x7f107000f6b0 > > 2015-03-25 15:07:26.517021 7f1058dd7700 20 dequeued request > > req=0x7f107000f0e0 > > 2015-03-25 15:07:26.517023 7f1058dd7700 20 RGWWQ: empty > > 2015-03-25 15:07:26.517081 7f1058dd7700 20 CONTENT_LENGTH=88 > > 2015-03-25 15:07:26.517084 7f1058dd7700 20 > > CONTENT_TYPE=application/octet-stream > > 2015-03-25 15:07:26.517085 7f1058dd7700 20 CONTEXT_DOCUMENT_ROOT=/var/www > > 2015-03-25 15:07:26.517086 7f1058dd7700 20 CONTEXT_PREFIX= > > 2015-03-25 15:07:26.517087 7f1058dd7700 20 DOCUMENT_ROOT=/var/www > > 2015-03-25 15:07:26.517088 7f1058dd7700 20 FCGI_ROLE=RESPONDER > > 2015-03-25 15:07:26.517089 7f1058dd7700 20 GATEWAY_INTERFACE=CGI/1.1 > > 2015-03-25 15:07:26.517090 7f1058dd7700 20 HTTP_AUTHORIZATION=AWS > > F79L68W19B3GCLOSE3F8:AcXqtvlBzBMpwdL+WuhDRoLT/Bs= > > 2015-03-25 15:07:26.517091 7f1058dd7700 20 HTTP_CONNECTION=Keep-Alive > > 2015-03-25 15:07:26.517092 7f1058dd7700 20 HTTP_DATE=Wed, 25 Mar 2015 > > 15:07:26 GMT > > 2015-03-25 15:07:26.517092 7f1058dd7700 20 HTTP_EXPECT=100-continue > > 2015-03-25 15:07:26.517093 7f1058dd7700 20 > > HTTP_HOST=test1.devops-os-cog01.devops.local > > 2015-03-25 15:07:26.517094 7f1058dd7700 20 > > HTTP_USER_AGENT=aws-sdk-java/unknown-version Windows_Server_2008_R2/6.1 > > Java_HotSpot(TM)_Client_VM/24.55-b03 > > 2015-03-25 15:07:26.517096 7f1058dd7700 20 > > HTTP_X_AMZ_META_CREATIONTIME=2015-03-25T15:07:26 > > 2015-03-25 15:07:26.517097 7f1058dd7700 20 HTTP_X_AMZ_META_SIZE=88 > > 2015-03-25 15:07:26.517098 7f1058dd7700 20 HTTP_X_AMZ_STORAGE_CLASS=STANDARD > > 2015-03-25 15:07:26.517099 7f1058dd7700 20 HTTPS=on > > 2015-03-25 15:07:26.517100 7f1058dd7700 20 > > PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > 2015-03-25 15:07:26.517100 7f1058dd7700 20 QUERY_STRING= > > 2015-03-25 15:07:26.517101 7f1058dd7700 20 REMOTE_ADDR=10.40.41.106 > > 2015-03-25 15:07:26.517102 7f1058dd7700 20 REMOTE_PORT=55439 > > 2015-03-25 15:07:26.517103 7f1058dd7700 20 REQUEST_METHOD=PUT > > 2015-03-25 15:07:26.517104 7f1058dd7700 20 REQUEST_SCHEME=https > > 2015-03-25 15:07:26.517105 7f1058dd7700 20 > > REQUEST_URI=/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 > > 2015-03-25 15:07:26.517106 7f1058dd7700 20 SCRIPT_FILENAME=/var/www/s3gw.fcgi > > 2015-03-25 15:07:26.517107 7f1058dd7700 20 > > SCRIPT_NAME=/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 > > 2015-03-25 15:07:26.517108 7f1058dd7700 20 > > SCRIPT_URI=https://test1.devops-os-cog01.devops.local/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 > > 2015-03-25 15:07:26.517109 7f1058dd7700 20 > > SCRIPT_URL=/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 > > 2015-03-25 15:07:26.517110 7f1058dd7700 20 SERVER_ADDR=10.40.41.64 > > 2015-03-25 15:07:26.517111 7f1058dd7700 20 SERVER_ADMIN=no-one@devops.local > > 2015-03-25 15:07:26.517112 7f1058dd7700 20 > > SERVER_NAME=test1.devops-os-cog01.devops.local > > 2015-03-25 15:07:26.517113 7f1058dd7700 20 SERVER_PORT=443 > > 2015-03-25 15:07:26.517114 7f1058dd7700 20 SERVER_PORT_SECURE=443 > > 2015-03-25 15:07:26.517115 7f1058dd7700 20 SERVER_PROTOCOL=HTTP/1.1 > > 2015-03-25 15:07:26.517116 7f1058dd7700 20 SERVER_SIGNATURE= > > 2015-03-25 15:07:26.517117 7f1058dd7700 20 SERVER_SOFTWARE=Apache/2.4.7 > > (Ubuntu) > > 2015-03-25 15:07:26.517119 7f1058dd7700 1 ====== starting new request > > req=0x7f107000f0e0 ===== > > 2015-03-25 15:07:26.517129 7f1058dd7700 2 req 6:0.000010::PUT > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3::initializing > > 2015-03-25 15:07:26.517132 7f1058dd7700 10 > > host=test1.devops-os-cog01.devops.local > > rgw_dns_name=devops-os-cog01.devops.local > > 2015-03-25 15:07:26.517143 7f1058dd7700 10 meta>> > > HTTP_X_AMZ_META_CREATIONTIME > > 2015-03-25 15:07:26.517147 7f1058dd7700 10 meta>> HTTP_X_AMZ_META_SIZE > > 2015-03-25 15:07:26.517150 7f1058dd7700 10 meta>> HTTP_X_AMZ_STORAGE_CLASS > > 2015-03-25 15:07:26.517155 7f1058dd7700 10 x>> > > x-amz-meta-creationtime:2015-03-25T15:07:26 > > 2015-03-25 15:07:26.517156 7f1058dd7700 10 x>> x-amz-meta-size:88 > > 2015-03-25 15:07:26.517157 7f1058dd7700 10 x>> x-amz-storage-class:STANDARD > > 2015-03-25 15:07:26.517167 7f1058dd7700 10 > > s->object=ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 s->bucket=test1 > > 2015-03-25 15:07:26.517171 7f1058dd7700 2 req 6:0.000052:s3:PUT > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3::getting op > > 2015-03-25 15:07:26.517175 7f1058dd7700 2 req 6:0.000055:s3:PUT > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3:put_obj:authorizing > > 2015-03-25 15:07:26.517209 7f1058dd7700 20 get_obj_state: rctx=0x7f106c025ce0 > > obj=.us-east.users:F79L68W19B3GCLOSE3F8 state=0x7f106c0260b8 > > s->prefetch_data=0 > > 2015-03-25 15:07:26.517215 7f1058dd7700 10 cache get: > > name=.us-east.users+F79L68W19B3GCLOSE3F8 : hit > > 2015-03-25 15:07:26.517219 7f1058dd7700 20 get_obj_state: s->obj_tag was set > > empty > > 2015-03-25 15:07:26.517224 7f1058dd7700 10 cache get: > > name=.us-east.users+F79L68W19B3GCLOSE3F8 : hit > > 2015-03-25 15:07:26.517248 7f1058dd7700 20 get_obj_state: rctx=0x7f106c025ce0 > > obj=.us-east.users.uid:test state=0x7f106c0267c8 s->prefetch_data=0 > > 2015-03-25 15:07:26.517252 7f1058dd7700 10 cache get: > > name=.us-east.users.uid+test : hit > > 2015-03-25 15:07:26.517256 7f1058dd7700 20 get_obj_state: s->obj_tag was set > > empty > > 2015-03-25 15:07:26.517261 7f1058dd7700 10 cache get: > > name=.us-east.users.uid+test : hit > > 2015-03-25 15:07:26.517296 7f1058dd7700 10 get_canon_resource(): > > dest=/test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 > > 2015-03-25 15:07:26.517298 7f1058dd7700 10 auth_hdr: > > PUT > > application/octet-stream > > Wed, 25 Mar 2015 15:07:26 GMT > > x-amz-meta-creationtime:2015-03-25T15:07:26 > > x-amz-meta-size:88 > > x-amz-storage-class:STANDARD > > /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 > > 2015-03-25 15:07:26.517341 7f1058dd7700 15 calculated > > digest=coYY2IBRECc42IXE7HQlYxBcqfU= > > 2015-03-25 15:07:26.517343 7f1058dd7700 15 > > auth_sign=AcXqtvlBzBMpwdL+WuhDRoLT/Bs= > > 2015-03-25 15:07:26.517344 7f1058dd7700 15 compare=-34 > > 2015-03-25 15:07:26.517346 7f1058dd7700 10 failed to authorize request > > 2015-03-25 15:07:26.517363 7f1058dd7700 2 req 6:0.000244:s3:PUT > > /ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3:put_obj:http status=403 > > 2015-03-25 15:07:26.517367 7f1058dd7700 1 ====== req done req=0x7f107000f0e0 > > http_status=403 ====== > > 2015-03-25 15:07:26.517374 7f1058dd7700 20 process_request() returned -1 > > 2015-03-25 15:07:28.058030 7f1088ff9700 2 > > RGWDataChangesLog::ChangesRenewThread: start > > Nothing pops up as being wrong. Assuming the first and second requests were done using the same credentials, I would guess that it might be the client incorrectly creating the canonical representation of the request. Maybe it doesn't use the x-amz-storage-class correctly there?. Can you try verifying what is the plain text signature string that it's using? Also, try disabling the storage class param, see if it makes any difference. > > Yehuda I've looked through the application logs but it doesn't appear to capture what the request string is that it's using. I also can't find out any way of disabling the x-amz-storage-class unfortunately. I did test the application against AWS S3 and the functionality I'm trying to use seems to work fine so it suggests it is a problem with my Ceph implementation. To help troubleshoot I've tried to capture some different requests for comparison: GET request from backup application (Successful): 2015-03-30 11:55:05.331145 7f103f5a4700 10 get_canon_resource(): dest=/test1/ 2015-03-30 11:55:05.331148 7f103f5a4700 10 auth_hdr: GET application/x-www-form-urlencoded; charset=utf-8 Mon, 30 Mar 2015 10:55:05 GMT /test1/ 2015-03-30 11:55:05.331195 7f103f5a4700 15 calculated digest=4tEUx7ERu+joNwYWrQboPhy/+Y4= 2015-03-30 11:55:05.331197 7f103f5a4700 15 auth_sign=4tEUx7ERu+joNwYWrQboPhy/+Y4= 2015-03-30 11:55:05.331198 7f103f5a4700 15 compare=0 2015-03-30 11:55:05.331200 7f103f5a4700 2 req 167:0.000252:s3:GET /:list_bucket:reading permissions 2015-03-30 11:55:05.331239 7f103f5a4700 20 get_obj_state: rctx=0x7f103f5a3230 obj=.us-east.domain.rgw:test1 state=0x7f109d479098 s->prefetch_data=0 2015-03-30 11:55:05.331245 7f103f5a4700 10 cache get: name=.us-east.domain.rgw+test1 : hit PUT request from backup application (Fails): 2015-03-30 11:55:05.414753 7f105cddf700 10 get_canon_resource(): dest=/test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 2015-03-30 11:55:05.414755 7f105cddf700 10 auth_hdr: PUT application/octet-stream Mon, 30 Mar 2015 10:55:05 GMT x-amz-meta-creationtime:2015-03-30T10:55:05 x-amz-meta-size:88 x-amz-storage-class:STANDARD /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3 2015-03-30 11:55:05.414800 7f105cddf700 15 calculated digest=DtPnePmBb00mfJgAr/z+uW70Unc= 2015-03-30 11:55:05.414802 7f105cddf700 15 auth_sign=iK1Wkt0uoIif9udGTf1P1fqF6Bk= 2015-03-30 11:55:05.414803 7f105cddf700 15 compare=37 2015-03-30 11:55:05.414805 7f105cddf700 10 failed to authorize request PUT request from "S3 Browser" application using S3 compatible connection (Successful): 2015-03-30 13:07:47.433691 7f10445ae700 10 get_canon_resource(): dest=/test1/february/?acl 2015-03-30 13:07:47.433694 7f10445ae700 10 auth_hdr: PUT x-amz-date:Mon, 30 Mar 2015 12:07:47 GMT /test1/february/?acl 2015-03-30 13:07:47.433740 7f10445ae700 15 calculated digest=9g91r6RZ3YCZawt+H4YuGaSlffQ= 2015-03-30 13:07:47.433742 7f10445ae700 15 auth_sign=9g91r6RZ3YCZawt+H4YuGaSlffQ= 2015-03-30 13:07:47.433743 7f10445ae700 15 compare=0 2015-03-30 13:07:47.433745 7f10445ae700 2 req 183:0.000227:s3:PUT /test1/february/:put_acls:reading permissions PUT request using boto S3 (Successful): 2015-03-30 13:26:29.394718 7f103f5a4700 10 get_canon_resource(): dest=/test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3/ 2015-03-30 13:26:29.394721 7f103f5a4700 10 auth_hdr: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Mon, 30 Mar 2015 12:26:02 GMT /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3/ 2015-03-30 13:26:29.394821 7f103f5a4700 15 calculated digest=ljvCTLAqySSfcJ3baWfK5776kJM= 2015-03-30 13:26:29.394823 7f103f5a4700 15 auth_sign=ljvCTLAqySSfcJ3baWfK5776kJM= 2015-03-30 13:26:29.394824 7f103f5a4700 15 compare=0 2015-03-30 13:26:29.394828 7f103f5a4700 2 req 188:0.000414:s3:PUT /test1/ca_ccifs_c6dccf63-ec57-45b2-87e7-d9b14b971ca3/:put_obj:reading permissions Does this help at all? Is it fair to assume it's not the application if it works ok with AWS S3? I'm wondering if there is a bug in the S3 implementation on Ceph which mean it handles one of these headers incorrectly: x-amz-meta-creationtime x-amz-meta-size x-amz-storage-class Thanks, Neville |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
