On Tue, Oct 3, 2023 at 9:06 AM Thomas Bennett <thomas@xxxxxxxx> wrote: > > Hi Jonas, > > Thanks :) that solved my issue. > > It would seem to me that this is heading towards something that the clients > s3 should paginate, but I couldn't find any documentation on how to > paginate bucket listings. the s3 ListBuckets API (https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html) doesn't support pagination, so there's no way for clients to do that but rgw itself should be able to paginate over the 'chunks' to return more than rgw_list_buckets_max_chunk entries in a single ListBuckets request. i opened a bug report for this at https://tracker.ceph.com/issues/63080 > All the information points to paginating object > listing - which makes sense. > > Just for competition of this thread: > > The rgw parameters are found at: Quincy radosgw config ref > <https://docs.ceph.com/en/quincy/radosgw/config-ref/> > > I ran the following command to update the parameter for all running rgw > daemons: > ceph config set client.rgw rgw_list_buckets_max_chunk 10000 > > And then confirmed the running daemons were configured: > ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx.asok config show | grep > rgw_list_buckets_max_chunk > "rgw_list_buckets_max_chunk": "10000", > > Kind regards, > Tom > > On Tue, 3 Oct 2023 at 13:30, Jonas Nemeiksis <jnemeiksis@xxxxxxxxx> wrote: > > > Hi, > > > > You should increase these default settings: > > > > rgw_list_buckets_max_chunk // for buckets > > rgw_max_listing_results // for objects > > > > On Tue, Oct 3, 2023 at 12:59 PM Thomas Bennett <thomas@xxxxxxxx> wrote: > > > >> Hi, > >> > >> I'm running a Ceph 17.2.5 Rados Gateway and I have a user with more than > >> 1000 buckets. > >> > >> When the client tries to list all their buckets using s3cmd, rclone and > >> python boto3, they all three only ever return the first 1000 bucket names. > >> I can confirm the buckets are all there (and more than 1000) by checking > >> with the radosgw-admin command. > >> > >> Have I missed a pagination limit for listing user buckets in the rados > >> gateway? > >> > >> Thanks, > >> Tom > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > > > > > > -- > > Jonas > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |