Hi Thomas, If I'm not mistaken, the RGW will paginate ListBuckets essentially like ListObjectsv1 if the S3 client provides the appropriate "marker" parameter values. COS does this too, I noticed. I'm not sure which S3 clients can be relied on to do this, though. Matt On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@xxxxxxxx> wrote: > Hi Jonas, > > Thanks :) that solved my issue. > > It would seem to me that this is heading towards something that the clients > s3 should paginate, but I couldn't find any documentation on how to > paginate bucket listings. All the information points to paginating object > listing - which makes sense. > > Just for competition of this thread: > > The rgw parameters are found at: Quincy radosgw config ref > <https://docs.ceph.com/en/quincy/radosgw/config-ref/> > > I ran the following command to update the parameter for all running rgw > daemons: > ceph config set client.rgw rgw_list_buckets_max_chunk 10000 > > And then confirmed the running daemons were configured: > ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep > rgw_list_buckets_max_chunk > "rgw_list_buckets_max_chunk": "10000", > > Kind regards, > Tom > > On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@xxxxxxxxx> wrote: > > > Hi, > > > > You should increase these default settings: > > > > rgw_list_buckets_max_chunk // for buckets > > rgw_max_listing_results // for objects > > > > On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@xxxxxxxx> wrote: > > > >> Hi, > >> > >> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than > >> 1000 buckets. > >> > >> When the client tries to list all their buckets using s3cmd, rclone and > >> python boto3, they all three only ever return the first 1000 bucket > names. > >> I can confirm the buckets are all there (and more than 1000) by checking > >> with the radosgw-admin command. > >> > >> Have I missed a pagination limit for listing user buckets in the rados > >> gateway? > >> > >> Thanks, > >> Tom > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > > > > > > -- > > Jonas > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |