Still, i'd strongly recommend sharding your big bucket before it gets much bigger. Typically it's during OSD recovery that you will encounter problems as it moves the index and locks all writes to it, it will start returning 500s. Problem is, you need to recreate the bucket before sharding it, it would be nice if RGW could shard an existing bucket's index. The check can take a long time. -Ben On Fri, Aug 28, 2015 at 9:16 AM, Sam Wouters <sam@xxxxxxxxx> wrote: > Hi, > > this bucket only has 13389 objects, so the index size shouldn't be a > problem. Also, on the same cluster we have an other bucket with 1200543 > objects (but no versioning configured), which has no issues. > > when we run a radosgw-admin bucket --check (--fix), nothing seems to be > happening. Putting an strace on the process shows a lot of lines like these: > [pid 99372] futex(0x2d730d4, FUTEX_WAIT_PRIVATE, 156619, NULL > <unfinished ...> > [pid 99385] futex(0x2da9410, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> > [pid 99371] futex(0x2da9410, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> > [pid 99385] <... futex resumed> ) = -1 EAGAIN (Resource > temporarily unavailable) > [pid 99371] <... futex resumed> ) = 0 > > but no errors in the ceph logs or health warnings. > > r, > Sam > > On 28-08-15 17:49, Ben Hines wrote: >> How many objects in the bucket? >> >> RGW has problems with index size once number of objects gets into the >> 900000+ level. The buckets need to be recreated with 'sharded bucket >> indexes' on: >> >> rgw override bucket index max shards = 23 >> >> You could also try repairing the index with: >> >> radosgw-admin bucket check --fix --bucket=<bucketname> >> >> -Ben >> >> On Fri, Aug 28, 2015 at 8:38 AM, Sam Wouters <sam@xxxxxxxxx> wrote: >>> Hi, >>> >>> we have a rgw bucket (with versioning) where PUT and GET operations for >>> specific objects succeed, but retrieving an object list fails. >>> Using python-boto, after a timeout just gives us an 500 internal error; >>> radosgw-admin just hangs. >>> Also a radosgw-admin bucket check just seems to hang... >>> >>> ceph version is 0.94.3 but this also was happening with 0.94.2, we >>> quietly hoped upgrading would fix but it didn't... >>> >>> r, >>> Sam >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |