Thanks! That seems to have been the issue. In general I really like ceph but it does seem like it has a few rough edges. We're currently using another distributed object storage that has been slightly unreliable and we've lost data twice now. That's why we're looking at alternatives and ceph is at the top of my list. What I understand is that cephs object storage is pretty stable and is used in production by some. We're only interested in the object storage/rados part so what's a more informed opinion on this than my speculation? Thanks again for your help! John On Thu, Jun 7, 2012 at 9:14 AM, Yehuda Sadeh <yehuda@xxxxxxxxxxx> wrote: > On Thu, Jun 7, 2012 at 12:03 AM, John Axel Eriksson <john@xxxxxxxxx> wrote: >> Hi I'm new to the list. >> >> We've been looking at Ceph as a possible replacement for our current >> distributed storage system. In particular >> we're interested in the object storage, so I started researching the >> radosgw. It did take me some time to get setup >> and the docs/wiki is missing lots of information (not pointing any >> fingers, just describing my experience). >> >> Anyway, I can't seem to actually do anything with the radosgw using >> any S3-client I've tried. The one I tried first >> was Fog (used in Chef for example). Fog is the "The Ruby cloud >> services library", available here https://github.com/fog/fog. >> Fog has support for many different cloud service providers. >> >> Anyway with Fog I get this in RadosGW logs: >> >> ############################## >> >> 2012-06-07 06:22:33.642827 7f638bfff700 2 req 4:0.000108:s3:GET /::getting op >> 2012-06-07 06:22:33.642831 7f638bfff700 2 req 4:0.000113:s3:GET >> /:list_buckets:authorizing >> 2012-06-07 06:22:33.642854 7f638bfff700 20 get_obj_state: >> rctx=0x7f637c0051e0 obj=.users:A state=0x7f637c005298 >> s->prefetch_data=0 >> 2012-06-07 06:22:33.642863 7f638bfff700 10 moving .users+A to cache LRU end >> 2012-06-07 06:22:33.642866 7f638bfff700 10 cache get: name=.users+A : hit >> 2012-06-07 06:22:33.642872 7f638bfff700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-06-07 06:22:33.642877 7f638bfff700 10 moving .users+A to cache LRU end >> 2012-06-07 06:22:33.642879 7f638bfff700 10 cache get: name=.users+A : hit >> 2012-06-07 06:22:33.642920 7f638bfff700 10 get_canon_resource(): dest=/ >> 2012-06-07 06:22:33.642923 7f638bfff700 10 auth_hdr: >> GET >> >> >> Thu, 07 Jun 2012 06:22:33 +0000 >> / >> 2012-06-07 06:22:33.642942 7f638bfff700 15 b64=zeK7YoQr63ajV6BRW6Z/UB1c4nM= >> 2012-06-07 06:22:33.642944 7f638bfff700 15 >> auth_sign=I9+abkuIjC89me95FoxmqApdrPs= >> 2012-06-07 06:22:33.642946 7f638bfff700 15 compare=-49 >> 2012-06-07 06:22:33.642949 7f638bfff700 10 failed to authorize request >> 2012-06-07 06:22:33.642954 7f638bfff700 10 --> Status: 403 >> >> So for some reason it fails to authorize the request. >> >> ############################### >> >> Then I moved to trying RightAws which is a different Cloud Services >> Library in Ruby. Here I get a little further (because I first get a >> 200 response): >> >> 2012-06-07 06:24:46.006867 7fde21d66700 10 cache get: name=.users+A : miss >> 2012-06-07 06:24:46.008245 7fde21d66700 10 cache put: name=.users+A >> 2012-06-07 06:24:46.008266 7fde21d66700 10 adding .users+A to cache LRU end >> 2012-06-07 06:24:46.008276 7fde21d66700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-06-07 06:24:46.008283 7fde21d66700 10 moving .users+A to cache LRU end >> 2012-06-07 06:24:46.008291 7fde21d66700 10 cache get: name=.users+A : >> type miss (requested=1, cached=6) >> 2012-06-07 06:24:46.008301 7fde21d66700 20 get_obj_state: >> rctx=0x7fde040055b0 obj=.users:A state=0x7fde040068b8 >> s->prefetch_data=0 >> 2012-06-07 06:24:46.008308 7fde21d66700 10 moving .users+A to cache LRU end >> 2012-06-07 06:24:46.008311 7fde21d66700 10 cache get: name=.users+A : hit >> 2012-06-07 06:24:46.008315 7fde21d66700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-06-07 06:24:46.008320 7fde21d66700 20 get_obj_state: >> rctx=0x7fde040055b0 obj=.users:A state=0x7fde040068b8 >> s->prefetch_data=0 >> 2012-06-07 06:24:46.008324 7fde21d66700 20 state for obj=.users:A is >> not atomic, not appending atomic test >> 2012-06-07 06:24:46.008327 7fde21d66700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-06-07 06:24:46.008864 7fde21d66700 20 rados->read r=0 bl.length=126 >> 2012-06-07 06:24:46.008898 7fde21d66700 10 cache put: name=.users+A >> 2012-06-07 06:24:46.008901 7fde21d66700 10 moving .users+A to cache LRU end >> 2012-06-07 06:24:46.008964 7fde21d66700 10 get_canon_resource(): dest=/ >> 2012-06-07 06:24:46.008968 7fde21d66700 10 auth_hdr: >> GET >> >> >> Thu, 07 Jun 2012 06:24:46 GMT >> / >> 2012-06-07 06:24:46.009059 7fde21d66700 15 b64=jpZmnXpUiWZq8IHBmsTKwKz4+Z4= >> 2012-06-07 06:24:46.009063 7fde21d66700 15 >> auth_sign=jpZmnXpUiWZq8IHBmsTKwKz4+Z4= >> 2012-06-07 06:24:46.009065 7fde21d66700 15 compare=0 >> 2012-06-07 06:24:46.009069 7fde21d66700 2 req 1:0.002478:s3:GET >> /:list_buckets:reading permissions >> 2012-06-07 06:24:46.009087 7fde21d66700 2 req 1:0.002496:s3:GET >> /:list_buckets:verifying op permissions >> 2012-06-07 06:24:46.009099 7fde21d66700 2 req 1:0.002509:s3:GET >> /:list_buckets:verifying op params >> 2012-06-07 06:24:46.009103 7fde21d66700 2 req 1:0.002513:s3:GET >> /:list_buckets:executing >> 2012-06-07 06:24:46.009750 7fde21d66700 10 --> Status: 200 >> >> 2012-06-07 06:24:46.009799 7fde21d66700 10 --> Content-Length: 318 >> >> 2012-06-07 06:24:46.009803 7fde21d66700 10 --> Accept-Ranges: bytes >> >> 2012-06-07 06:24:46.009807 7fde21d66700 10 --> Content-type: application/xml >> >> 2012-06-07 06:24:46.009825 7fde21d66700 5 nothing to log for operation >> 2012-06-07 06:24:46.009832 7fde21d66700 2 req 1:0.003241:s3:GET >> /:list_buckets:http status=200 >> >> And now I do a PUT: >> >> 2012-06-07 06:24:46.018316 7fde097fa700 2 req 2:0.000171:s3:PUT >> /johnkey::getting op >> 2012-06-07 06:24:46.018322 7fde097fa700 2 req 2:0.000177:s3:PUT >> /johnkey:create_bucket:authorizing >> 2012-06-07 06:24:46.018346 7fde097fa700 20 get_obj_state: >> rctx=0x7fddfc0058a0 obj=.users:A state=0x7fddfc005958 >> s->prefetch_data=0 >> 2012-06-07 06:24:46.018356 7fde097fa700 10 moving .users+A to cache LRU end >> 2012-06-07 06:24:46.018360 7fde097fa700 10 cache get: name=.users+A : >> type miss (requested=6, cached=3) >> 2012-06-07 06:24:46.019922 7fde097fa700 10 cache put: name=.users+A >> 2012-06-07 06:24:46.019931 7fde097fa700 10 moving .users+A to cache LRU end >> 2012-06-07 06:24:46.019940 7fde097fa700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-06-07 06:24:46.019948 7fde097fa700 10 moving .users+A to cache LRU end >> 2012-06-07 06:24:46.019951 7fde097fa700 10 cache get: name=.users+A : hit >> 2012-06-07 06:24:46.019999 7fde097fa700 10 get_canon_resource(): dest=/johnkey >> 2012-06-07 06:24:46.020003 7fde097fa700 10 auth_hdr: >> PUT >> >> >> Thu, 07 Jun 2012 06:24:46 GMT >> /johnkey >> 2012-06-07 06:24:46.020026 7fde097fa700 15 b64=RPIY36txOP127RgLYseHC8Ua9kc= >> 2012-06-07 06:24:46.020028 7fde097fa700 15 >> auth_sign=h020Ewm8NbfLYebWsXWkWCB6cDI= >> 2012-06-07 06:24:46.020030 7fde097fa700 15 compare=22 >> 2012-06-07 06:24:46.020035 7fde097fa700 10 failed to authorize request >> 2012-06-07 06:24:46.020041 7fde097fa700 10 --> Status: 403 >> >> ################################# >> >> I've also tried s3cmd on the commandline which didn't work either. As >> far as I've understood radosgw is supposed to be compatible with S3. >> > > It looks like the clients you're using are trying to access buckets > via the virtual host api: it preprends the bucket name to the service > host name. This requires wildcard dns, and also to have the 'rgw dns > name' config option correctly set in your ceph.conf. > > Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
