Re: RGW resharding

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I think the shard number recommendation is 100K objects/per shard/per
bucket. If you have many objects but they are spread in many
buckets/containers and each bucket/container have less than 1.6M
objects(max_shards=16) then you should be ok.

linyunfan

Adrian Nicolae <adrian.nicolae@xxxxxxxxxx> 于2020年5月25日周一 下午3:04写道:
>
> I'm using only Swift , not S3.  We have a container for every customer.
> Right now there are thousands of containers.
>
>
>
> On 5/25/2020 9:02 AM, lin yunfan wrote:
> > Can you store your data in different buckets?
> >
> > linyunfan
> >
> > Adrian Nicolae <adrian.nicolae@xxxxxxxxxx> 于2020年5月19日周二 下午3:32写道:
> >> Hi,
> >>
> >> I have the following Ceph Mimic setup :
> >>
> >> - a bunch of old servers with 3-4 SATA drives each (74 OSDs in total)
> >>
> >> - index/leveldb is stored on each OSD (so no SSD drives, just SATA)
> >>
> >> - the current usage  is :
> >>
> >> GLOBAL:
> >>       SIZE        AVAIL       RAW USED     %RAW USED
> >>       542 TiB     105 TiB      437 TiB         80.67
> >> POOLS:
> >>       NAME                           ID     USED        %USED MAX
> >> AVAIL     OBJECTS
> >>       .rgw.root                      1      1.1 KiB         0 26
> >> TiB            4
> >>       default.rgw.control            2          0 B         0 26
> >> TiB            8
> >>       default.rgw.meta               3       20 MiB         0 26
> >> TiB        75357
> >>       default.rgw.log                4          0 B         0 26
> >> TiB         4271
> >>       default.rgw.buckets.data       5      290 TiB     85.05 51 TiB
> >> 78067284
> >>       default.rgw.buckets.non-ec     6          0 B         0 26
> >> TiB            0
> >>       default.rgw.buckets.index      7          0 B         0 26
> >> TiB       603008
> >>
> >> - rgw_override_bucket_index_max_shards = 16.   Clients are accessing RGW
> >> via Swift, not S3.
> >>
> >> - the replication schema is EC 4+2.
> >>
> >> We are using this Ceph cluster as  a secondary storage for another
> >> storage infrastructure (which is more expensive) and we are offloading
> >> cold data (big files with a low number of downloads/reads from our
> >> customer). This way we can lower the TCO .  So most of the files are big
> >> ( a few GB at least).
> >>
> >>    So far Ceph is doing well considering that I don't have big
> >> expectations from current hardware.  I'm a bit worried however that we
> >> have 78 M objects with max_shards=16 and we will probably reach 100M in
> >> the next few months. Do I need a increase the max shards to ensure the
> >> stability of the cluster ?  I read that storing more than 1 M of objects
> >> in a single bucket can lead to OSD's flapping or having io timeouts
> >> during deep-scrub or even to have ODS's failures due to the leveldb
> >> compacting all the time if we have a large number of DELETEs.
> >>
> >> Any advice would be appreciated.
> >>
> >>
> >> Thank you,
> >>
> >> Adrian Nicolae
> >>
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users@xxxxxxx
> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux