On Mon, Oct 21, 2019 at 11:20 AM Arash Shams <ara4sh@xxxxxxxxxxx> wrote: > Yes listing v2 is not supported yet. I checked metadata osds and all of them are 600gb 10k hdd I dont think this was the issue. > I will test the --allow-unordered 5 million objects in a single bucket and metadata on HDD is a disaster waiting to happen if this continues to grow. rgw setups with large buckets need SSDs (or better NVMe) for metadata if you value availability. Recovering after a node failure will be horrible if you keep this on HDDs. Paul > > > Regards > ________________________________ > From: Paul Emmerich <paul.emmerich@xxxxxxxx> > Sent: Thursday, October 17, 2019 10:00 AM > To: Arash Shams <ara4sh@xxxxxxxxxxx> > Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> > Subject: Re: RadosGW cant list objects when there are too many of them > > Listing large buckets is slow due to S3 ordering requirements, it's > approximately O(n^2). > However, I wouldn't consider 5M to be a large bucket, it should go to > only ~50 shards which should still perform reasonable. How fast are > your metadata OSDs? > > Try --allow-unordered in radosgw-admin to get an unordered result > which is only O(n) as you'd expect. > > For boto3: I'm not sure if v2 object listing is available yet (I think > it has only been merged into master but has not yet made it into a > release?). It doesn't support unordered listing but there has been > some work to implement it there, not sure about the current state. > > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh@xxxxxxxxxxx> wrote: > > > > Dear All > > > > I have a bucket with 5 million Objects and I cant list objects with > > radosgw-admin bucket list --bucket=bucket | jq .[].name > > or listing files using boto3 > > > > s3 = boto3.client('s3', > > endpoint_url=credentials['endpoint_url'], > > aws_access_key_id=credentials['access_key'], > > aws_secret_access_key=credentials['secret_key']) > > > > response = s3.list_objects_v2(Bucket=bucket_name) > > for item in response['Contents']: > > print(item['Key']) > > > > what is the solution ? how can I find list of my objects ? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |