When you say that you can't list it with boto or radosgw-admin, what
happens? Does it give you an error, or just hang/timeout? How many
shards does the bucket have?
On 10/17/19 6:00 AM, Paul Emmerich wrote:
Listing large buckets is slow due to S3 ordering requirements, it's
approximately O(n^2).
However, I wouldn't consider 5M to be a large bucket, it should go to
only ~50 shards which should still perform reasonable. How fast are
your metadata OSDs?
I just wanted to share that recent work by Eric and Mark is showing huge
improvements in sharded listing performance:
https://github.com/ceph/ceph/pull/30853#issuecomment-541964967
Try --allow-unordered in radosgw-admin to get an unordered result
which is only O(n) as you'd expect.
For boto3: I'm not sure if v2 object listing is available yet (I think
it has only been merged into master but has not yet made it into a
release?). It doesn't support unordered listing but there has been
some work to implement it there, not sure about the current state.
Paul
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
