And the debug log because that last log was obviously not helpful...
2014-03-12 23:57:49.497780 7ff97e7dd700 1 ====== starting new request req=0x23bc650 =====
2014-03-12 23:57:49.498198 7ff97e7dd700 2 req 1:0.000419::PUT /test::initializing
2014-03-12 23:57:49.498233 7ff97e7dd700 10 host=s3.amazonaws.com rgw_dns_name=us-west-1.domain
2014-03-12 23:57:49.498366 7ff97e7dd700 10 s->object=<NULL> s->bucket=test
2014-03-12 23:57:49.498437 7ff97e7dd700 2 req 1:0.000659:s3:PUT /test::getting op
2014-03-12 23:57:49.498448 7ff97e7dd700 2 req 1:0.000670:s3:PUT /test:create_bucket:authorizing
2014-03-12 23:57:49.498508 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : miss
2014-03-12 23:57:49.500852 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH
2014-03-12 23:57:49.500865 7ff97e7dd700 10 adding .us-west-1.users+BLAHBLAHBLAH to cache LRU end
2014-03-12 23:57:49.500886 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end
2014-03-12 23:57:49.500889 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : type miss (requested=1, cached=6)
2014-03-12 23:57:49.500907 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end
2014-03-12 23:57:49.500910 7ff97e7dd700 10 cache get: name=.us-west-1.users+BLAHBLAHBLAH : hit
2014-03-12 23:57:49.502663 7ff97e7dd700 10 cache put: name=.us-west-1.users+BLAHBLAHBLAH
2014-03-12 23:57:49.502667 7ff97e7dd700 10 moving .us-west-1.users+BLAHBLAHBLAH to cache LRU end
2014-03-12 23:57:49.502700 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : miss
2014-03-12 23:57:49.505128 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test
2014-03-12 23:57:49.505138 7ff97e7dd700 10 adding .us-west-1.users.uid+test to cache LRU end
2014-03-12 23:57:49.505157 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end
2014-03-12 23:57:49.505160 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : type miss (requested=1, cached=6)
2014-03-12 23:57:49.505176 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end
2014-03-12 23:57:49.505178 7ff97e7dd700 10 cache get: name=.us-west-1.users.uid+test : hit
2014-03-12 23:57:49.507401 7ff97e7dd700 10 cache put: name=.us-west-1.users.uid+test
2014-03-12 23:57:49.507406 7ff97e7dd700 10 moving .us-west-1.users.uid+test to cache LRU end
2014-03-12 23:57:49.507521 7ff97e7dd700 10 get_canon_resource(): dest=/test
2014-03-12 23:57:49.507529 7ff97e7dd700 10 auth_hdr:
PUT
binary/octet-stream
Wed, 12 Mar 2014 23:57:51 GMT
/test
2014-03-12 23:57:49.507674 7ff97e7dd700 2 req 1:0.009895:s3:PUT /test:create_bucket:reading permissions
2014-03-12 23:57:49.507682 7ff97e7dd700 2 req 1:0.009904:s3:PUT /test:create_bucket:verifying op mask
2014-03-12 23:57:49.507695 7ff97e7dd700 2 req 1:0.009917:s3:PUT /test:create_bucket:verifying op permissions
2014-03-12 23:57:49.509604 7ff97e7dd700 2 req 1:0.011826:s3:PUT /test:create_bucket:verifying op params
2014-03-12 23:57:49.509615 7ff97e7dd700 2 req 1:0.011836:s3:PUT /test:create_bucket:executing
2014-03-12 23:57:49.509694 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+test : miss
2014-03-12 23:57:49.512229 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+test
2014-03-12 23:57:49.512259 7ff97e7dd700 10 adding .us-west-1.domain.rgw+test to cache LRU end
2014-03-12 23:57:49.512333 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : miss
2014-03-12 23:57:49.518216 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+.pools.avail
2014-03-12 23:57:49.518228 7ff97e7dd700 10 adding .us-west-1.domain.rgw+.pools.avail to cache LRU end
2014-03-12 23:57:49.518248 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end
2014-03-12 23:57:49.518251 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : type miss (requested=1, cached=6)
2014-03-12 23:57:49.518270 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end
2014-03-12 23:57:49.518272 7ff97e7dd700 10 cache get: name=.us-west-1.domain.rgw+.pools.avail : hit
2014-03-12 23:57:49.520295 7ff97e7dd700 10 cache put: name=.us-west-1.domain.rgw+.pools.avail
2014-03-12 23:57:49.520348 7ff97e7dd700 10 moving .us-west-1.domain.rgw+.pools.avail to cache LRU end
2014-03-12 23:57:49.522672 7ff97e7dd700 2 req 1:0.024893:s3:PUT /test:create_bucket:http status=403
2014-03-12 23:57:49.523204 7ff97e7dd700 1 ====== req done req=0x23bc650 http_status=403 ======
On Wed, Mar 12, 2014 at 7:36 PM, Greg Poirier <greg.poirier@xxxxxxxxxx> wrote:
The saga continues...So, after fiddling with haproxy a bit, I managed to make sure that my requests were hitting the RADOS Gateway.NOW, I get a 403 from my ruby script:2014-03-12 23:34:08.289670 7fda9bfbf700 1 ====== starting new request req=0x215a780 =====2014-03-12 23:34:08.305105 7fda9bfbf700 1 ====== req done req=0x215a780 http_status=403 ======The aws-s3 gem forces the Host header to be set to s3.amazonaws.com -- and I am wondering if this could potentially cause a problem. Is that the case? Or is radosgw able to just know where it is supposed to look for objects based on its configuration? I assume the latter, otherwise we would have multi-tenant radosgw instead of needing to have an instance per zone.So does it ignore the Host header?Is this 403 related to my problems with the user error I found earlier where I'm unable to view the user with the radosgw-admin tool?On Wed, Mar 12, 2014 at 1:54 PM, Greg Poirier <greg.poirier@xxxxxxxxxx> wrote:
Also... what are linger_ops?ceph --admin-daemon /var/run/ceph/ceph-client.radosgw.<hostname>.asok objecter_requests{ "ops": [],"linger_ops": [{ "linger_id": 1,"pg": "7.4322fa9f","osd": 25,"object_id": "notify.0","object_locator": "@7","snapid": "head","registering": "head","registered": "head"},{ "linger_id": 2,"pg": "7.16dafda0","osd": 132,"object_id": "notify.1","object_locator": "@7","snapid": "head","registering": "head","registered": "head"},{ "linger_id": 3,"pg": "7.88aa5c95","osd": 32,"object_id": "notify.2","object_locator": "@7","snapid": "head","registering": "head","registered": "head"},{ "linger_id": 4,"pg": "7.f8c99aee","osd": 62,"object_id": "notify.3","object_locator": "@7","snapid": "head","registering": "head","registered": "head"},{ "linger_id": 5,"pg": "7.a204812d","osd": 129,"object_id": "notify.4","object_locator": "@7","snapid": "head","registering": "head","registered": "head"},{ "linger_id": 6,"pg": "7.31099063","osd": 28,"object_id": "notify.5","object_locator": "@7","snapid": "head","registering": "head","registered": "head"},{ "linger_id": 7,"pg": "7.97c520d4","osd": 135,"object_id": "notify.6","object_locator": "@7","snapid": "head","registering": "head","registered": "head"},{ "linger_id": 8,"pg": "7.84ada7c9","osd": 94,"object_id": "notify.7","object_locator": "@7","snapid": "head","registering": "head","registered": "head"}],"pool_ops": [],"pool_stat_ops": [],"statfs_ops": [],"command_ops": []}On Wed, Mar 12, 2014 at 10:45 AM, Greg Poirier <greg.poirier@xxxxxxxxxx> wrote:
Rados GW and Ceph versions installed:Version: 0.67.7-1preciseI create a user:radosgw-admin --name client.radosgw.<hostname> user create --uid test --display-name "Test User"
It outputs some JSON that looks convincing:{ "user_id": "test","display_name": "test user","email": "","suspended": 0,"max_buckets": 1000,"auid": 0,"subusers": [],"keys": [{ "user": "test","access_key": "<snip>","secret_key": "<snip>"},{ "user": "test","access_key": "<snip>","secret_key": "<snip>"}],"swift_keys": [],"caps": [],"op_mask": "read, write, delete","default_placement": "","placement_tags": []}There are two keys because I have tried this twice.I can see it in metadata list:radosgw-admin --name client.radosgw.<hostname> metadata list user["test","us-east-2","us-west-1"]I then try to get user info:radosgw-admin --name client.radosgw.<hostname> user info testcould not fetch user info: no user info savedI try to create a bucket with the user using Ruby's aws/s3 API:equire 'aws/s3'AWS::S3::Base.establish_connection!(access_key_id: '<snip>',secret_access_key: '<snip>',use_ssl: true,server: '<snip>',persistent: true)AWS::S3::Bucket.create('test')file = 'sloth.txt'AWS::S3::S3Object.store(file, open(file), 'test')bucket = AWS::S3::Bucket.find('test')puts bucketbucket.each do |object|puts "#{object.key}\t#{object.about['content-length']}\t#{object.about['last-modified']}"endAnd I get the following:#<AWS::S3::Bucket:0x007f9852854e58>/Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:235:in `method_missing': undefined local variable or method `name' for #<AWS::S3::Bucket:0x007f9852854e58> (NameError)from /Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/bucket.rb:313:in `reload!'from /Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/bucket.rb:242:in `objects'from /Users/greg.poirier/.rvm/gems/ruby-1.9.3-p429/gems/aws-s3-0.6.3/lib/aws/s3/bucket.rb:253:in `each'from test.rb:21:in `<main>'The bucket fails to be created:radosgw-admin --name client.radosgw.<hostname> bucket list[]And also this:radosgw-admin --name client.radosgw.<hostname> metadata list bucket[]2014-03-12 17:42:42.221112 7f426b779780 -1 failed to list objects pool_iterate returned r=-2So clearly there is something going on here. My questions:Is this failure to create a bucket related to do the "no user info saved" error?What would cause the "no user info saved" error?What may be causing the bucket to not be created?
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
