radosgw bucket listing (s3 ls s3://$bucketname) slow with ~2 billion objects

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



One of our radosgw buckets has grown a lot in size, `rgw bucket stats --bucket $bucketname` reports a total of 2,110,269,538 objects with the bucket index sharded across 32768 shards, listing the root context of the bucket with `s3 ls s3://$bucketname` takes more than an hour which is the hard limit to first-byte on our nginx reverse proxy and the aws-cli times out long before that timeout limit is hit.

The software we use supports sharding the data across multiple s3 buckets but before I go ahead and enable this, has anyone ever had that many objects in a single RGW bucket and can let me know how you solved the problem of RGW taking a long time to read the full index?

-- 
Best regards

Katie Holly
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux