Can you bump up debug (debug rgw = 20, debug ms = 1), and see if the operations (bucket listing and bucket check) go into some kind of infinite loop? Yehuda On Tue, Sep 1, 2015 at 1:16 AM, Sam Wouters <sam@xxxxxxxxx> wrote: > Hi, I've started the bucket --check --fix on friday evening and it's > still running. 'ceph -s' shows the cluster health as OK, I don't know if > there is anything else I could check? Is there a way of finding out if > its actually doing something? > > We only have this issue on the one bucket with versioning enabled, I > can't get rid of the feeling it has something todo with that. The > "underscore bug" is also still present on that bucket > (http://tracker.ceph.com/issues/12819). Not sure if thats related in any > way. > Are there any alternatives, as for example copy all the objects into a > new bucket without versioning? Simple way would be to list the objects > and copy them to a new bucket, but bucket listing is not working so... > > -Sam > > > On 31-08-15 10:47, Gregory Farnum wrote: >> This generally shouldn't be a problem at your bucket sizes. Have you >> checked that the cluster is actually in a healthy state? The sleeping >> locks are normal but should be getting woken up; if they aren't it >> means the object access isn't working for some reason. A down PG or >> something would be the simplest explanation. >> -Greg >> >> On Fri, Aug 28, 2015 at 6:52 PM, Sam Wouters <sam@xxxxxxxxx> wrote: >>> Ok, maybe I'm to impatient. It would be great if there were some verbose >>> or progress logging of the radosgw-admin tool. >>> I will start a check and let it run over the weekend. >>> >>> tnx, >>> Sam >>> >>> On 28-08-15 18:16, Sam Wouters wrote: >>>> Hi, >>>> >>>> this bucket only has 13389 objects, so the index size shouldn't be a >>>> problem. Also, on the same cluster we have an other bucket with 1200543 >>>> objects (but no versioning configured), which has no issues. >>>> >>>> when we run a radosgw-admin bucket --check (--fix), nothing seems to be >>>> happening. Putting an strace on the process shows a lot of lines like these: >>>> [pid 99372] futex(0x2d730d4, FUTEX_WAIT_PRIVATE, 156619, NULL >>>> <unfinished ...> >>>> [pid 99385] futex(0x2da9410, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> >>>> [pid 99371] futex(0x2da9410, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> >>>> [pid 99385] <... futex resumed> ) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> [pid 99371] <... futex resumed> ) = 0 >>>> >>>> but no errors in the ceph logs or health warnings. >>>> >>>> r, >>>> Sam >>>> >>>> On 28-08-15 17:49, Ben Hines wrote: >>>>> How many objects in the bucket? >>>>> >>>>> RGW has problems with index size once number of objects gets into the >>>>> 900000+ level. The buckets need to be recreated with 'sharded bucket >>>>> indexes' on: >>>>> >>>>> rgw override bucket index max shards = 23 >>>>> >>>>> You could also try repairing the index with: >>>>> >>>>> radosgw-admin bucket check --fix --bucket=<bucketname> >>>>> >>>>> -Ben >>>>> >>>>> On Fri, Aug 28, 2015 at 8:38 AM, Sam Wouters <sam@xxxxxxxxx> wrote: >>>>>> Hi, >>>>>> >>>>>> we have a rgw bucket (with versioning) where PUT and GET operations for >>>>>> specific objects succeed, but retrieving an object list fails. >>>>>> Using python-boto, after a timeout just gives us an 500 internal error; >>>>>> radosgw-admin just hangs. >>>>>> Also a radosgw-admin bucket check just seems to hang... >>>>>> >>>>>> ceph version is 0.94.3 but this also was happening with 0.94.2, we >>>>>> quietly hoped upgrading would fix but it didn't... >>>>>> >>>>>> r, >>>>>> Sam >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |