|
Thanks paul
Yes listing v2 is not supported yet. I checked metadata osds and all of them are 600gb 10k hdd I dont think this was the issue.
I will test the --allow-unordered
Regards
From: Paul Emmerich <paul.emmerich@xxxxxxxx>
Sent: Thursday, October 17, 2019 10:00 AM To: Arash Shams <ara4sh@xxxxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Subject: Re: RadosGW cant list objects when there are too many of them Listing large buckets is slow due to S3 ordering requirements, it's
approximately O(n^2). However, I wouldn't consider 5M to be a large bucket, it should go to only ~50 shards which should still perform reasonable. How fast are your metadata OSDs? Try --allow-unordered in radosgw-admin to get an unordered result which is only O(n) as you'd expect. For boto3: I'm not sure if v2 object listing is available yet (I think it has only been merged into master but has not yet made it into a release?). It doesn't support unordered listing but there has been some work to implement it there, not sure about the current state. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Oct 17, 2019 at 9:19 AM Arash Shams <ara4sh@xxxxxxxxxxx> wrote: > > Dear All > > I have a bucket with 5 million Objects and I cant list objects with > radosgw-admin bucket list --bucket=bucket | jq .[].name > or listing files using boto3 > > s3 = boto3.client('s3', > endpoint_url=credentials['endpoint_url'], > aws_access_key_id=credentials['access_key'], > aws_secret_access_key=credentials['secret_key']) > > response = s3.list_objects_v2(Bucket=bucket_name) > for item in response['Contents']: > print(item['Key']) > > what is the solution ? how can I find list of my objects ? > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx |
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
