Dear ceph users! I've been running into some issues trying to use the radosgw S3 API for talking to my ceph cluster. For avoiding carrying away some issues from older installations, I reinstalled everything from ground up using 0.79 and 0.80.1, running on Debian and Ubuntu as well, with the exactly same results using Nginx or Apache as a fastcgi gateway. It seems to be related to some failing permissions I can't put my finger on, since users digest and tokens do match. The following is a radosgw log for 'create bucket' operation using the python boto S3 library on python2.7, which throws a 403 since failed to perform the operation ("S3ResponseError: S3ResponseError: 403 Forbidden"), but I get the same issue with the s3cmd tool as well. 2014-05-20 11:34:08.630807 7f7853fff700 20 enqueued request req=0x7f784c00f050 2014-05-20 11:34:08.630826 7f7853fff700 20 RGWWQ: 2014-05-20 11:34:08.630827 7f7853fff700 20 req: 0x7f784c00f050 2014-05-20 11:34:08.630831 7f7853fff700 10 allocated request req=0x7f784c00f340 2014-05-20 11:34:08.630837 7f785a7fc700 20 dequeued request req=0x7f784c00f050 2014-05-20 11:34:08.630845 7f785a7fc700 20 RGWWQ: empty 2014-05-20 11:34:08.630884 7f785a7fc700 20 CONTENT_LENGTH=0 2014-05-20 11:34:08.630885 7f785a7fc700 20 CONTEXT_DOCUMENT_ROOT=/var/www/ 2014-05-20 11:34:08.630885 7f785a7fc700 20 CONTEXT_PREFIX= 2014-05-20 11:34:08.630886 7f785a7fc700 20 DOCUMENT_ROOT=/var/www/ 2014-05-20 11:34:08.630889 7f785a7fc700 20 FCGI_ROLE=RESPONDER 2014-05-20 11:34:08.630891 7f785a7fc700 20 GATEWAY_INTERFACE=CGI/1.1 2014-05-20 11:34:08.630892 7f785a7fc700 20 HTTP_ACCEPT_ENCODING=identity 2014-05-20 11:34:08.630892 7f785a7fc700 20 HTTP_AUTHORIZATION=AWS 5DFF8DCDXPK2AJ557N3J:F/exUe4uIjwJHRZC6+3MNPOBnIU= 2014-05-20 11:34:08.630893 7f785a7fc700 20 HTTP_DATE=Tue, 20 May 2014 14:34:20 GMT 2014-05-20 11:34:08.630893 7f785a7fc700 20 HTTP_HOST=serverIP 2014-05-20 11:34:08.630894 7f785a7fc700 20 HTTP_USER_AGENT=Boto/2.27.0 Python/2.7.6 Linux/3.14-1-amd64 2014-05-20 11:34:08.630894 7f785a7fc700 20 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2014-05-20 11:34:08.630895 7f785a7fc700 20 QUERY_STRING=page=my-test-bucket-1¶ms=/ 2014-05-20 11:34:08.630895 7f785a7fc700 20 REMOTE_ADDR=clientIP 2014-05-20 11:34:08.630896 7f785a7fc700 20 REMOTE_PORT=50588 2014-05-20 11:34:08.630896 7f785a7fc700 20 REQUEST_METHOD=PUT 2014-05-20 11:34:08.630897 7f785a7fc700 20 REQUEST_SCHEME=http 2014-05-20 11:34:08.630897 7f785a7fc700 20 REQUEST_URI=/my-test-bucket-1/ 2014-05-20 11:34:08.630898 7f785a7fc700 20 RGW_LOG_LEVEL=20 2014-05-20 11:34:08.630898 7f785a7fc700 20 RGW_PRINT_CONTINUE=yes 2014-05-20 11:34:08.630899 7f785a7fc700 20 RGW_SHOULD_LOG=yes 2014-05-20 11:34:08.630899 7f785a7fc700 20 SCRIPT_FILENAME=/var/www/s3gw.fcgi 2014-05-20 11:34:08.630900 7f785a7fc700 20 SCRIPT_NAME=/my-test-bucket-1/ 2014-05-20 11:34:08.630900 7f785a7fc700 20 SCRIPT_URI=http://serverIP/my-test-bucket-1/ 2014-05-20 11:34:08.630902 7f785a7fc700 20 SCRIPT_URL=/my-test-bucket-1/ 2014-05-20 11:34:08.630903 7f785a7fc700 20 SERVER_ADDR=serverIP 2014-05-20 11:34:08.630903 7f785a7fc700 20 SERVER_ADMIN=webmaster at localhost 2014-05-20 11:34:08.630904 7f785a7fc700 20 SERVER_NAME=serverIP 2014-05-20 11:34:08.630904 7f785a7fc700 20 SERVER_PORT=80 2014-05-20 11:34:08.630905 7f785a7fc700 20 SERVER_PROTOCOL=HTTP/1.1 2014-05-20 11:34:08.630905 7f785a7fc700 20 SERVER_SIGNATURE=<address>Apache/2.4.7 (Ubuntu) Server at serverIP Port 80</address> 2014-05-20 11:34:08.630906 7f785a7fc700 20 SERVER_SOFTWARE=Apache/2.4.7 (Ubuntu) 2014-05-20 11:34:08.630907 7f785a7fc700 1 ====== starting new request req=0x7f784c00f050 ===== 2014-05-20 11:34:08.630917 7f785a7fc700 2 req 13:0.000010::PUT /my-test-bucket-1/::initializing 2014-05-20 11:34:08.630920 7f785a7fc700 10 host=serverIP rgw_dns_name=labs.mydomain.com 2014-05-20 11:34:08.630945 7f785a7fc700 10 s->object=<NULL> s->bucket=my-test-bucket-1 2014-05-20 11:34:08.630948 7f785a7fc700 2 req 13:0.000041:s3:PUT /my-test-bucket-1/::getting op *2014-05-20 11:34:08.630952 7f785a7fc700 2 req 13:0.000045:s3:PUT /my-test-bucket-1/:create_bucket:authorizing* 2014-05-20 11:34:08.630979 7f785a7fc700 20 get_obj_state: rctx=0x7f780c0059f0 obj=.users:5DFF8DCDXPK2AJ557N3J state=0x7f780c005f18 s->prefetch_data=0 2014-05-20 11:34:08.630984 7f785a7fc700 10 cache get: name=.users+5DFF8DCDXPK2AJ557N3J : hit 2014-05-20 11:34:08.630988 7f785a7fc700 20 get_obj_state: s->obj_tag was set empty 2014-05-20 11:34:08.630992 7f785a7fc700 10 cache get: name=.users+5DFF8DCDXPK2AJ557N3J : hit 2014-05-20 11:34:08.631011 7f785a7fc700 20 get_obj_state: rctx=0x7f780c0059f0 obj=.users.uid:test123 state=0x7f780c006938 s->prefetch_data=0 2014-05-20 11:34:08.631014 7f785a7fc700 10 cache get: name=.users.uid+test123 : hit 2014-05-20 11:34:08.631016 7f785a7fc700 20 get_obj_state: s->obj_tag was set empty 2014-05-20 11:34:08.631018 7f785a7fc700 10 cache get: name=.users.uid+test123 : hit 2014-05-20 11:34:08.631047 7f785a7fc700 10 get_canon_resource(): dest=/my-test-bucket-1/ 2014-05-20 11:34:08.631049 7f785a7fc700 10 auth_hdr: PUT Tue, 20 May 2014 14:34:20 GMT /my-test-bucket-1/ *2014-05-20 11:34:08.631083 7f785a7fc700 15 calculated digest=F/exUe4uIjwJHRZC6+3MNPOBnIU=** **2014-05-20 11:34:08.631084 7f785a7fc700 15 auth_sign=F/exUe4uIjwJHRZC6+3MNPOBnIU=** **2014-05-20 11:34:08.631085 7f785a7fc700 15 compare=0* 2014-05-20 11:34:08.631087 7f785a7fc700 2 req 13:0.000180:s3:PUT /my-test-bucket-1/:create_bucket:reading permissions 2014-05-20 11:34:08.631089 7f785a7fc700 2 req 13:0.000182:s3:PUT /my-test-bucket-1/:create_bucket:init op 2014-05-20 11:34:08.631090 7f785a7fc700 2 req 13:0.000183:s3:PUT /my-test-bucket-1/:create_bucket:verifying op mask 2014-05-20 11:34:08.631092 7f785a7fc700 20 required_mask= 2 user.op_mask=7 2014-05-20 11:34:08.631093 7f785a7fc700 2 req 13:0.000185:s3:PUT /my-test-bucket-1/:create_bucket:verifying op permissions 2014-05-20 11:34:08.631138 7f785a7fc700 1 -- serverIP:0/1010944 --> cluster_member3:6800/11860 -- osd_op(client.5728.0:1791 test123.buckets [call user.list_buckets] 7.4add1db5 ack+read e239) v4 -- ?+0 0x7f780c007470 con 0x7f78a9c5af40 2014-05-20 11:34:08.633253 7f789e7d6700 1 -- serverIP:0/1010944 <== osd.2 cluster_member3:6800/11860 1009 ==== osd_op_reply(1791 test123.buckets [call] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 182+0+0 (1020662873 0 0) 0x7f7880004d60 con 0x7f78a9c5af40 2014-05-20 11:34:08.633326 7f785a7fc700 2 req 13:0.002419:s3:PUT /my-test-bucket-1/:create_bucket:verifying op params 2014-05-20 11:34:08.633338 7f785a7fc700 2 req 13:0.002431:s3:PUT /my-test-bucket-1/:create_bucket:executing 2014-05-20 11:34:08.633371 7f785a7fc700 20 get_obj_state: rctx=0x7f785a7fb230 obj=.rgw:my-test-bucket-1 state=0x7f780c007a88 s->prefetch_data=0 2014-05-20 11:34:08.633378 7f785a7fc700 10 cache get: name=.rgw+my-test-bucket-1 : type miss (requested=22, cached=0) 2014-05-20 11:34:08.633414 7f785a7fc700 1 -- serverIP:0/1010944 --> cluster_member3:6800/11860 -- osd_op(client.5728.0:1792 my-test-bucket-1 [call version.read,getxattrs,stat] 5.347aa3b6 ack+read e239) v4 -- ?+0 0x7f780c00a3e0 con 0x7f78a9c5af40 2014-05-20 11:34:08.634978 7f789e7d6700 1 -- serverIP:0/1010944 <== osd.2 cluster_member3:6800/11860 1010 ==== osd_op_reply(1792 my-test-bucket-1 [call,getxattrs,stat] v0'0 uv0 ack = -2 (No such file or directory)) v6 ==== 267+0+0 (1067375874 0 0) 0x7f7880003a70 con 0x7f78a9c5af40 2014-05-20 11:34:08.635043 7f785a7fc700 10 cache put: name=.rgw+my-test-bucket-1 2014-05-20 11:34:08.635051 7f785a7fc700 10 moving .rgw+my-test-bucket-1 to cache LRU end 2014-05-20 11:34:08.635086 7f785a7fc700 1 -- serverIP:0/1010944 --> serverIP:6789/0 -- pool_op(create pool 0 auid 0 tid 1793 name .rgw.buckets.index v239) v4 -- ?+0 0x7f780c007450 con 0x7f78a9c58660 *2014-05-20 11:34:08.635293 7f789e7d6700 1 -- serverIP:0/1010944 <== mon.1 serverIP:6789/0 456 ==== pool_op_reply(tid 1793 (1) Operation not permitted v239) v1 ==== 43+0+0 (767193955 0 0) 0x7f78940016f0 con 0x7f78a9c58660* *2014-05-20 11:34:08.635326 7f785a7fc700 20 rgw_create_bucket returned ret=-1 bucket=my-test-bucket-1(@{i=.rgw.buckets.index}.rgw.buckets[default.5728.6])* *2014-05-20 11:34:08.635353 7f785a7fc700 2 req 13:0.004446:s3:PUT /my-test-bucket-1/:create_bucket:http status=403* *2014-05-20 11:34:08.635360 7f785a7fc700 1 ====== req done req=0x7f784c00f050 http_status=403 ======* 2014-05-20 11:34:08.635367 7f785a7fc700 20 process_request() returned -1 Configuration is pretty standard crafted by ceph-deploy: [client.radosgw.gw0] user = www-data host = radosgwHost0 keyring = /etc/ceph/keyring.radosgw.gw0 rgw_socket_path = /var/run/ceph/radosgw0.sock log_file = /var/log/ceph/radosgw.log debug_rgw = 20 debug_ms = 1 rgw_dns_name = labs.mydomain.com Any ideas on what I'm doing wrong or what should I put my eyes into? I've exhausted all the possible ideas I had, so I'll be very grateful to explore new ones! Thanks in advance! Cheers, \d -- BOFH excuse #450: Terrorists crashed an airplane into the server room, have to remove /bin/laden. (rm -rf /bin/laden) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140520/9e6e8f2c/attachment.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140520/9e6e8f2c/attachment.pgp>