On Tue, Sep 11, 2012 at 6:48 PM, Yehuda Sadeh <yehuda@xxxxxxxxxxx> wrote:
> On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@xxxxxxxxxxx> wrote:
>> On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@xxxxxxxxx> wrote:
>>> Every acl operation ending with 403 in PUT.
>>>
>>> ~# s3 -u test oc
>>> Bucket Status
>>> -------------------------------------------------------- --------------------
>>> oc Access Denied
>>>
>>> Anyone know why, and how to enable this bucket ?? Now i have problems
>>> with cluster, because there is no way to upload new file
>>>
>>> ~# s3 -u getacl oc
>>>
>>> ERROR: ErrorAccessDenied
>>>
>>
>> User somehow lost bucket ownership (was it actually the owner?). Do
>> you know how to reproduce the issue? any remaining logs?
>>
>> Try getting bucket info:
>>
>> # radosgw-admin bucket stats --bucket=oc
>>
>> If that doesn't fail and actually shows relevant info, try checking
>> whether the user credentials match the s3 tool credentials.
>>
> Oh, and thinking about it some more.. 'oc' is a too short name for a
> bucket (requires min of 3 chars). How did you create it? The failure
> may be related.
Yes i made a shortcut of name :))
Right now every bucket in pool, are afected
:~#radosgw-admin bucket stats --bucket=lvstest
{ "bucket": "lvstest",
"pool": ".rgw.buckets",
"id": "1142048.1",
"marker": "1142048.1",
"owner": "0",
"usage": { "rgw.main": { "size_kb": 1,
"size_kb_actual": 4,
"num_objects": 1}}}
:~# radosgw-admin bucket stats --bucket=ocdn
{ "bucket": "ocdn",
"pool": ".rgw.buckets",
"id": "4168.2",
"marker": "4168.2",
"owner": "0",
"usage": { "rgw.main": { "size_kb": 513059717,
"size_kb_actual": 516402364,
"num_objects": 1606730}}}
Credentials from radosgw-admin user info match that from clients requests.
Every GET, PUT, HEAD using this credentials works fine, but only one
operations does not work (403 from radosgw) - setting acl for object
for a public-read. Setting canned acl with PUT for public-read from
s3lib work good, but get/set acl failed.
list bucket object works good, and list buckets via s3 client.
Now i can't reproduce, but i will dig logs from radosgw, for related
time, when this happend.
Example 403 from radosgw log, before that PUT of object ends with 200:
2012-09-11 19:36:34.346312 7fb25d7fa700 1 ====== req done
req=0x1435980 http_status=403 ======
2012-09-11 19:37:04.342894 7fb25d7fa700 20 dequeued request req=0x13994c0
2012-09-11 19:37:04.342903 7fb25d7fa700 20 RGWWQ: empty
2012-09-11 19:37:04.342910 7fb25d7fa700 1 ====== starting new request
req=0x13994c0 =====
2012-09-11 19:37:04.342948 7fb25d7fa700 2 req 39665:0.000038::::initializing
2012-09-11 19:37:04.342971 7fb25d7fa700 10
s->object=images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg
s->bucket=ocdn
2012-09-11 19:37:04.342983 7fb25d7fa700 10 meta>> HTTP_X_AMZ_ACL=public-read
2012-09-11 19:37:04.342991 7fb25d7fa700 10 x>> x-amz-acl:public-read
2012-09-11 19:37:04.342996 7fb25d7fa700 20 FCGI_ROLE=RESPONDER
2012-09-11 19:37:04.342997 7fb25d7fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi
2012-09-11 19:37:04.342999 7fb25d7fa700 20 QUERY_STRING=acl
2012-09-11 19:37:04.343001 7fb25d7fa700 20 REQUEST_METHOD=PUT
2012-09-11 19:37:04.343002 7fb25d7fa700 20 CONTENT_TYPE=
2012-09-11 19:37:04.343003 7fb25d7fa700 20 CONTENT_LENGTH=0
2012-09-11 19:37:04.343004 7fb25d7fa700 20 HTTP_CONTENT_LENGTH=0
2012-09-11 19:37:04.343005 7fb25d7fa700 20
SCRIPT_NAME=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg
2012-09-11 19:37:04.343006 7fb25d7fa700 20
REQUEST_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg
2012-09-11 19:37:04.343007 7fb25d7fa700 20
DOCUMENT_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg
2012-09-11 19:37:04.343008 7fb25d7fa700 20 DOCUMENT_ROOT=/var/www
2012-09-11 19:37:04.343009 7fb25d7fa700 20 SERVER_PROTOCOL=HTTP/1.1
2012-09-11 19:37:04.343010 7fb25d7fa700 20 GATEWAY_INTERFACE=CGI/1.1
2012-09-11 19:37:04.343011 7fb25d7fa700 20 SERVER_SOFTWARE=nginx/1.2.0
2012-09-11 19:37:04.343012 7fb25d7fa700 20 REMOTE_ADDR=10.177.62.9
2012-09-11 19:37:04.343013 7fb25d7fa700 20 REMOTE_PORT=56378
2012-09-11 19:37:04.343014 7fb25d7fa700 20 SERVER_ADDR=10.177.0.3
2012-09-11 19:37:04.343015 7fb25d7fa700 20 SERVER_PORT=80
2012-09-11 19:37:04.343016 7fb25d7fa700 20 SERVER_NAME=
2012-09-11 19:37:04.343017 7fb25d7fa700 20 REDIRECT_STATUS=200
2012-09-11 19:37:04.343018 7fb25d7fa700 20 RGW_SHOULD_LOG=no
2012-09-11 19:37:04.343019 7fb25d7fa700 20 HTTP_HOST=10.177.0.3
2012-09-11 19:37:04.343020 7fb25d7fa700 20 HTTP_ACCEPT_ENCODING=identity
2012-09-11 19:37:04.343021 7fb25d7fa700 20 HTTP_DATE=Tue, 11 Sep 2012
17:37:02 GMT
2012-09-11 19:37:04.343022 7fb25d7fa700 20 HTTP_X_AMZ_ACL=public-read
2012-09-11 19:37:04.343023 7fb25d7fa700 20 HTTP_AUTHORIZATION=AWS
73VF66Q2JH5PT5K7QJ3A:mh7yBIFGPtdI1MAjW39/1ywV580=
2012-09-11 19:37:04.343024 7fb25d7fa700 20 HTTP_USER_AGENT=Boto/2.5.2 (linux2)
2012-09-11 19:37:04.343026 7fb25d7fa700 2 req 39665:0.000116:s3:PUT
/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg::getting
op
2012-09-11 19:37:04.343031 7fb25d7fa700 2 req 39665:0.000121:s3:PUT
/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:authorizing
2012-09-11 19:37:04.343047 7fb25d7fa700 20 get_obj_state:
rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A
state=0x7fb23c00b468 s->prefetch_data=0
2012-09-11 19:37:04.344527 7fb25d7fa700 20 get_obj_state: s->obj_tag
was set empty
2012-09-11 19:37:04.344542 7fb25d7fa700 20 get_obj_state:
rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A
state=0x7fb23c00b468 s->prefetch_data=0
2012-09-11 19:37:04.345847 7fb25d7fa700 20 get_obj_state: s->obj_tag
was set empty
2012-09-11 19:37:04.345858 7fb25d7fa700 20 get_obj_state:
rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A
state=0x7fb23c00b468 s->prefetch_data=0
2012-09-11 19:37:04.345863 7fb25d7fa700 20 state for
obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic
test
2012-09-11 19:37:04.345866 7fb25d7fa700 20 rados->read obj-ofs=0
read_ofs=0 read_len=16384
2012-09-11 19:37:04.347069 7fb25d7fa700 20 rados->read r=0 bl.length=231
2012-09-11 19:37:04.347121 7fb25d7fa700 10 get_canon_resource():
dest=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg
2012-09-11 19:37:04.347126 7fb25d7fa700 10 auth_hdr:
2012-09-11 19:37:04.347195 7fb25d7fa700 15 b64=9qJQCGmPbTXcT6a4qD3kED0PPdY=
2012-09-11 19:37:04.347198 7fb25d7fa700 15
auth_sign=mh7yBIFGPtdI1MAjW39/1ywV580=
2012-09-11 19:37:04.347199 7fb25d7fa700 15 compare=52
2012-09-11 19:37:04.347201 7fb25d7fa700 10 failed to authorize request
2012-09-11 19:37:04.347208 7fb25d7fa700 10 --> Status: 403
2012-09-11 19:37:04.347218 7fb25d7fa700 10 --> Content-Length: 78
2012-09-11 19:37:04.347220 7fb25d7fa700 10 --> Accept-Ranges: bytes
2012-09-11 19:37:04.347223 7fb25d7fa700 10 --> Content-type: application/xml
2012-09-11 19:37:04.347346 7fb25d7fa700 2 req 39665:0.004436:s3:PUT
/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:http
status=403
2012-09-11 19:37:04.347600 7fb25d7fa700 1 ====== req done
req=0x13994c0 http_status=403 ======
2012-09-11 19:40:04.354709 7fb25d7fa700 20 dequeued request req=0x13994c0
Interesting is that second cluster, which is replicated via s3 client,
get the same issue, from that in production one. Is this can be
generated by many error on radosgw layer ??
I will try to find beginning of this problems in logs.
>
> Yehuda
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
