Thanks for all the responses, much appreciated. Upping the chunk size fixes my problem in the short term but I upgrade to 17.2.6 :) Kind regards, Tom On Tue, 3 Oct 2023 at 15:28, Matt Benjamin <mbenjami@xxxxxxxxxx> wrote: > Hi Thomas, > > If I'm not mistaken, the RGW will paginate ListBuckets essentially like > ListObjectsv1 if the S3 client provides the appropriate "marker" parameter > values. COS does this too, I noticed. I'm not sure which S3 clients can > be relied on to do this, though. > > Matt > > On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@xxxxxxxx> wrote: > >> Hi Jonas, >> >> Thanks :) that solved my issue. >> >> It would seem to me that this is heading towards something that the >> clients >> s3 should paginate, but I couldn't find any documentation on how to >> paginate bucket listings. All the information points to paginating object >> listing - which makes sense. >> >> Just for competition of this thread: >> >> The rgw parameters are found at: Quincy radosgw config ref >> <https://docs.ceph.com/en/quincy/radosgw/config-ref/> >> >> I ran the following command to update the parameter for all running rgw >> daemons: >> ceph config set client.rgw rgw_list_buckets_max_chunk 10000 >> >> And then confirmed the running daemons were configured: >> ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep >> rgw_list_buckets_max_chunk >> "rgw_list_buckets_max_chunk": "10000", >> >> Kind regards, >> Tom >> >> On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@xxxxxxxxx> >> wrote: >> >> > Hi, >> > >> > You should increase these default settings: >> > >> > rgw_list_buckets_max_chunk // for buckets >> > rgw_max_listing_results // for objects >> > >> > On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@xxxxxxxx> wrote: >> > >> >> Hi, >> >> >> >> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more >> than >> >> 1000 buckets. >> >> >> >> When the client tries to list all their buckets using s3cmd, rclone and >> >> python boto3, they all three only ever return the first 1000 bucket >> names. >> >> I can confirm the buckets are all there (and more than 1000) by >> checking >> >> with the radosgw-admin command. >> >> >> >> Have I missed a pagination limit for listing user buckets in the rados >> >> gateway? >> >> >> >> Thanks, >> >> Tom >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users@xxxxxxx >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> > >> > >> > -- >> > Jonas >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |