The numbers you're reporting strike me as surprising as well. Which version are you running? In case you're not aware, listing of buckets is not a very efficient operation given that the listing is required to return with objects in lexical order. They are distributed across the shards via a hash, which is not in lexical order. So every shard has to have a chunk read and brought to the rgw and the top elements are sorted and returned. For example, in order to return the first 1000 object names, it asks each of the 32 shards for their first 1000 object names, and then does a selection process to get the first 1000 among the 32000. It returns that, and the process is then repeated. I'm unfamiliar with your mc script/command, so I don't know if that might be contributing to the issue. We have added the ability to list buckets in unsorted order and made that accessible via s3 and swift and that's been backported all the way to upstream luminous. Eric On 9/26/18 5:27 PM, Graham Allan wrote: > I have one user bucket, where inexplicably (to me), the bucket takes an > eternity to list, though only on the top level. There are two > subfolders, each of which lists individually at a completely normal > speed... > > eg (using minio client): > > > [~] % time ./mc ls fried/friedlab/ > > [2018-09-26 16:15:48 CDT] 0B impute/ > > [2018-09-26 16:15:48 CDT] 0B wgs/ > > > > real 1m59.390s > > > > [~] % time ./mc ls -r fried/friedlab/ > > ... > > real 3m18.013s > > > > [~] % time ./mc ls -r fried/friedlab/impute > > ... > > real 0m13.512s > > > > [~] % time ./mc ls -r fried/friedlab/wgs > > ... > > real 0m6.437s > > The bucket has about 55k objects total, with 32 index shards on a > replicated ssd pool. It shouldn't be taking this long but I can't > imagine what could be causing this. I haven't found any others behaving > this way. I'd think it has to be some problem with the bucket index, but > what...? > > I did naively try some "radosgw-admin bucket check [--fix]" commands > with no change. > > Graham _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |