Re: RGW bucket sharding in Jewel

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



There have been some bugs in the past with dynamic resharding, but those seem to be the exception not the norm.  I was surprised to find that some of our buckets had resharded themselves from 12 shards in Jewel to over 200 shards in luminous without me even realizing it.  We would have resharded in Jewel, but were on a version where it was bugged and it wasn't possible without upgrading.  When I went back to reshard later after the upgrade to luminous I found that the buckets had all resharded themselves without any problems.  When we have blocked requests on our nvme osds (where the RGW metadata lives) I always check if any resharding is happening, but it never is.  I haven't tracked any problems down to resharding.

On Tue, Jun 19, 2018 at 3:44 PM Matt Benjamin <mbenjami@xxxxxxxxxx> wrote:
The increased time to list sharded buckets is currently expected, yes.
In turn other operations such as put and delete should be faster in
proportion to two factors, the number of shards on independent PGs
(serialization by PG), and the spread of shards onto independent OSD
devices (speedup from scaling onto more OSD devices, presuming
available iops on those devices).

New bucket index formats are coming in future to help listing
workloads.  Also, as of recent master (and probably Jewel and Luminous
at this point, modulo some latency for the backports) we have added an
"allow-unordered" option to S3 and Swift listing arguments that should
remove the penalty from sharding.  This causes results to be returned
in partial order, rather than the total order most applications
expect.

Matt

On Tue, Jun 19, 2018 at 9:34 AM, Matthew Vernon <mv3@xxxxxxxxxxxx> wrote:
> Hi,
>
> Some of our users have Quite Large buckets (up to 20M objects in a
> bucket), and AIUI best practice would be to have sharded indexes for
> those buckets (of the order of 1 shard per 100k objects).
>
> On a trivial test case (make a 1M-object bucket, shard index to 10
> shards, s3cmd ls s3://bucket >/dev/null), sharding makes the bucket
> listing slower (not a lot, but a bit).
>
> Are there simple(ish) workflows we could use to demonstrate an
> improvement from index sharding?
>
> Thanks,
>
> Matthew
>
> [I understand that Luminous has dynamic resharding, but it seems a bit
> unstable for production use; is that still the case?]
>
>
> --
>  The Wellcome Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>



--

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux