[radosgw] unable to perform any operation using s3 api

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Dear ceph users!

I've been running into some issues trying to use the radosgw S3 API for
talking to my ceph cluster.
For avoiding carrying away some issues from older installations, I
reinstalled everything from ground up using 0.79 and 0.80.1, running on
Debian and Ubuntu as well, with the exactly same results using Nginx or
Apache as a fastcgi gateway.

It seems to be related to some failing permissions I can't put my finger
on, since users digest and tokens do match.
The following is a radosgw log for 'create bucket' operation using the
python boto S3 library on python2.7, which throws a 403 since failed to
perform the operation ("S3ResponseError: S3ResponseError: 403
Forbidden"), but I get the same issue with the s3cmd tool as well.


2014-05-20 11:34:08.630807 7f7853fff700 20 enqueued request
req=0x7f784c00f050
2014-05-20 11:34:08.630826 7f7853fff700 20 RGWWQ:
2014-05-20 11:34:08.630827 7f7853fff700 20 req: 0x7f784c00f050
2014-05-20 11:34:08.630831 7f7853fff700 10 allocated request
req=0x7f784c00f340
2014-05-20 11:34:08.630837 7f785a7fc700 20 dequeued request
req=0x7f784c00f050
2014-05-20 11:34:08.630845 7f785a7fc700 20 RGWWQ: empty
2014-05-20 11:34:08.630884 7f785a7fc700 20 CONTENT_LENGTH=0
2014-05-20 11:34:08.630885 7f785a7fc700 20 CONTEXT_DOCUMENT_ROOT=/var/www/
2014-05-20 11:34:08.630885 7f785a7fc700 20 CONTEXT_PREFIX=
2014-05-20 11:34:08.630886 7f785a7fc700 20 DOCUMENT_ROOT=/var/www/
2014-05-20 11:34:08.630889 7f785a7fc700 20 FCGI_ROLE=RESPONDER
2014-05-20 11:34:08.630891 7f785a7fc700 20 GATEWAY_INTERFACE=CGI/1.1
2014-05-20 11:34:08.630892 7f785a7fc700 20 HTTP_ACCEPT_ENCODING=identity
2014-05-20 11:34:08.630892 7f785a7fc700 20 HTTP_AUTHORIZATION=AWS
5DFF8DCDXPK2AJ557N3J:F/exUe4uIjwJHRZC6+3MNPOBnIU=
2014-05-20 11:34:08.630893 7f785a7fc700 20 HTTP_DATE=Tue, 20 May 2014
14:34:20 GMT
2014-05-20 11:34:08.630893 7f785a7fc700 20 HTTP_HOST=serverIP
2014-05-20 11:34:08.630894 7f785a7fc700 20 HTTP_USER_AGENT=Boto/2.27.0
Python/2.7.6 Linux/3.14-1-amd64
2014-05-20 11:34:08.630894 7f785a7fc700 20
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2014-05-20 11:34:08.630895 7f785a7fc700 20
QUERY_STRING=page=my-test-bucket-1&params=/
2014-05-20 11:34:08.630895 7f785a7fc700 20 REMOTE_ADDR=clientIP
2014-05-20 11:34:08.630896 7f785a7fc700 20 REMOTE_PORT=50588
2014-05-20 11:34:08.630896 7f785a7fc700 20 REQUEST_METHOD=PUT
2014-05-20 11:34:08.630897 7f785a7fc700 20 REQUEST_SCHEME=http
2014-05-20 11:34:08.630897 7f785a7fc700 20 REQUEST_URI=/my-test-bucket-1/
2014-05-20 11:34:08.630898 7f785a7fc700 20 RGW_LOG_LEVEL=20
2014-05-20 11:34:08.630898 7f785a7fc700 20 RGW_PRINT_CONTINUE=yes
2014-05-20 11:34:08.630899 7f785a7fc700 20 RGW_SHOULD_LOG=yes
2014-05-20 11:34:08.630899 7f785a7fc700 20
SCRIPT_FILENAME=/var/www/s3gw.fcgi
2014-05-20 11:34:08.630900 7f785a7fc700 20 SCRIPT_NAME=/my-test-bucket-1/
2014-05-20 11:34:08.630900 7f785a7fc700 20
SCRIPT_URI=http://serverIP/my-test-bucket-1/
2014-05-20 11:34:08.630902 7f785a7fc700 20 SCRIPT_URL=/my-test-bucket-1/
2014-05-20 11:34:08.630903 7f785a7fc700 20 SERVER_ADDR=serverIP
2014-05-20 11:34:08.630903 7f785a7fc700 20 SERVER_ADMIN=webmaster at localhost
2014-05-20 11:34:08.630904 7f785a7fc700 20 SERVER_NAME=serverIP
2014-05-20 11:34:08.630904 7f785a7fc700 20 SERVER_PORT=80
2014-05-20 11:34:08.630905 7f785a7fc700 20 SERVER_PROTOCOL=HTTP/1.1
2014-05-20 11:34:08.630905 7f785a7fc700 20
SERVER_SIGNATURE=<address>Apache/2.4.7 (Ubuntu) Server at serverIP Port
80</address>

2014-05-20 11:34:08.630906 7f785a7fc700 20 SERVER_SOFTWARE=Apache/2.4.7
(Ubuntu)
2014-05-20 11:34:08.630907 7f785a7fc700  1 ====== starting new request
req=0x7f784c00f050 =====
2014-05-20 11:34:08.630917 7f785a7fc700  2 req 13:0.000010::PUT
/my-test-bucket-1/::initializing
2014-05-20 11:34:08.630920 7f785a7fc700 10 host=serverIP
rgw_dns_name=labs.mydomain.com
2014-05-20 11:34:08.630945 7f785a7fc700 10 s->object=<NULL>
s->bucket=my-test-bucket-1
2014-05-20 11:34:08.630948 7f785a7fc700  2 req 13:0.000041:s3:PUT
/my-test-bucket-1/::getting op
*2014-05-20 11:34:08.630952 7f785a7fc700  2 req 13:0.000045:s3:PUT
/my-test-bucket-1/:create_bucket:authorizing*
2014-05-20 11:34:08.630979 7f785a7fc700 20 get_obj_state:
rctx=0x7f780c0059f0 obj=.users:5DFF8DCDXPK2AJ557N3J state=0x7f780c005f18
s->prefetch_data=0
2014-05-20 11:34:08.630984 7f785a7fc700 10 cache get:
name=.users+5DFF8DCDXPK2AJ557N3J : hit
2014-05-20 11:34:08.630988 7f785a7fc700 20 get_obj_state: s->obj_tag was
set empty
2014-05-20 11:34:08.630992 7f785a7fc700 10 cache get:
name=.users+5DFF8DCDXPK2AJ557N3J : hit
2014-05-20 11:34:08.631011 7f785a7fc700 20 get_obj_state:
rctx=0x7f780c0059f0 obj=.users.uid:test123 state=0x7f780c006938
s->prefetch_data=0
2014-05-20 11:34:08.631014 7f785a7fc700 10 cache get:
name=.users.uid+test123 : hit
2014-05-20 11:34:08.631016 7f785a7fc700 20 get_obj_state: s->obj_tag was
set empty
2014-05-20 11:34:08.631018 7f785a7fc700 10 cache get:
name=.users.uid+test123 : hit
2014-05-20 11:34:08.631047 7f785a7fc700 10 get_canon_resource():
dest=/my-test-bucket-1/
2014-05-20 11:34:08.631049 7f785a7fc700 10 auth_hdr:
PUT


Tue, 20 May 2014 14:34:20 GMT
/my-test-bucket-1/
*2014-05-20 11:34:08.631083 7f785a7fc700 15 calculated
digest=F/exUe4uIjwJHRZC6+3MNPOBnIU=**
**2014-05-20 11:34:08.631084 7f785a7fc700 15
auth_sign=F/exUe4uIjwJHRZC6+3MNPOBnIU=**
**2014-05-20 11:34:08.631085 7f785a7fc700 15 compare=0*
2014-05-20 11:34:08.631087 7f785a7fc700  2 req 13:0.000180:s3:PUT
/my-test-bucket-1/:create_bucket:reading permissions
2014-05-20 11:34:08.631089 7f785a7fc700  2 req 13:0.000182:s3:PUT
/my-test-bucket-1/:create_bucket:init op
2014-05-20 11:34:08.631090 7f785a7fc700  2 req 13:0.000183:s3:PUT
/my-test-bucket-1/:create_bucket:verifying op mask
2014-05-20 11:34:08.631092 7f785a7fc700 20 required_mask= 2 user.op_mask=7
2014-05-20 11:34:08.631093 7f785a7fc700  2 req 13:0.000185:s3:PUT
/my-test-bucket-1/:create_bucket:verifying op permissions
2014-05-20 11:34:08.631138 7f785a7fc700  1 -- serverIP:0/1010944 -->
cluster_member3:6800/11860 -- osd_op(client.5728.0:1791 test123.buckets
[call user.list_buckets] 7.4add1db5 ack+read e239) v4 -- ?+0
0x7f780c007470 con 0x7f78a9c5af40
2014-05-20 11:34:08.633253 7f789e7d6700  1 -- serverIP:0/1010944 <==
osd.2 cluster_member3:6800/11860 1009 ==== osd_op_reply(1791
test123.buckets [call] v0'0 uv0 ack = -2 (No such file or directory)) v6
==== 182+0+0 (1020662873 0 0) 0x7f7880004d60 con 0x7f78a9c5af40
2014-05-20 11:34:08.633326 7f785a7fc700  2 req 13:0.002419:s3:PUT
/my-test-bucket-1/:create_bucket:verifying op params
2014-05-20 11:34:08.633338 7f785a7fc700  2 req 13:0.002431:s3:PUT
/my-test-bucket-1/:create_bucket:executing
2014-05-20 11:34:08.633371 7f785a7fc700 20 get_obj_state:
rctx=0x7f785a7fb230 obj=.rgw:my-test-bucket-1 state=0x7f780c007a88
s->prefetch_data=0
2014-05-20 11:34:08.633378 7f785a7fc700 10 cache get:
name=.rgw+my-test-bucket-1 : type miss (requested=22, cached=0)
2014-05-20 11:34:08.633414 7f785a7fc700  1 -- serverIP:0/1010944 -->
cluster_member3:6800/11860 -- osd_op(client.5728.0:1792 my-test-bucket-1
[call version.read,getxattrs,stat] 5.347aa3b6 ack+read e239) v4 -- ?+0
0x7f780c00a3e0 con 0x7f78a9c5af40
2014-05-20 11:34:08.634978 7f789e7d6700  1 -- serverIP:0/1010944 <==
osd.2 cluster_member3:6800/11860 1010 ==== osd_op_reply(1792
my-test-bucket-1 [call,getxattrs,stat] v0'0 uv0 ack = -2 (No such file
or directory)) v6 ==== 267+0+0 (1067375874 0 0) 0x7f7880003a70 con
0x7f78a9c5af40
2014-05-20 11:34:08.635043 7f785a7fc700 10 cache put:
name=.rgw+my-test-bucket-1
2014-05-20 11:34:08.635051 7f785a7fc700 10 moving .rgw+my-test-bucket-1
to cache LRU end
2014-05-20 11:34:08.635086 7f785a7fc700  1 -- serverIP:0/1010944 -->
serverIP:6789/0 -- pool_op(create pool 0 auid 0 tid 1793 name
.rgw.buckets.index v239) v4 -- ?+0 0x7f780c007450 con 0x7f78a9c58660
*2014-05-20 11:34:08.635293 7f789e7d6700  1 -- serverIP:0/1010944 <==
mon.1 serverIP:6789/0 456 ==== pool_op_reply(tid 1793 (1) Operation not
permitted v239) v1 ==== 43+0+0 (767193955 0 0) 0x7f78940016f0 con
0x7f78a9c58660*
*2014-05-20 11:34:08.635326 7f785a7fc700 20 rgw_create_bucket returned
ret=-1
bucket=my-test-bucket-1(@{i=.rgw.buckets.index}.rgw.buckets[default.5728.6])*
*2014-05-20 11:34:08.635353 7f785a7fc700  2 req 13:0.004446:s3:PUT
/my-test-bucket-1/:create_bucket:http status=403*
*2014-05-20 11:34:08.635360 7f785a7fc700  1 ====== req done
req=0x7f784c00f050 http_status=403 ======*
2014-05-20 11:34:08.635367 7f785a7fc700 20 process_request() returned -1

Configuration is pretty standard crafted by ceph-deploy:
[client.radosgw.gw0]
user = www-data
host = radosgwHost0
keyring = /etc/ceph/keyring.radosgw.gw0
rgw_socket_path = /var/run/ceph/radosgw0.sock
log_file = /var/log/ceph/radosgw.log
debug_rgw = 20
debug_ms = 1
rgw_dns_name = labs.mydomain.com


Any ideas on what I'm doing wrong or what should I put my eyes into?
I've exhausted all the possible ideas I had, so I'll be very grateful to
explore new ones!

Thanks in advance!


Cheers,

\d

-- 
BOFH excuse #450:
Terrorists crashed an airplane into the server room, have to remove /bin/laden. (rm -rf /bin/laden)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140520/9e6e8f2c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140520/9e6e8f2c/attachment.pgp>
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux