Re: RGW resharding

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I'm using only Swift , not S3.  We have a container for every customer. Right now there are thousands of containers.



On 5/25/2020 9:02 AM, lin yunfan wrote:
Can you store your data in different buckets?

linyunfan

Adrian Nicolae <adrian.nicolae@xxxxxxxxxx> 于2020年5月19日周二 下午3:32写道:
Hi,

I have the following Ceph Mimic setup :

- a bunch of old servers with 3-4 SATA drives each (74 OSDs in total)

- index/leveldb is stored on each OSD (so no SSD drives, just SATA)

- the current usage  is :

GLOBAL:
      SIZE        AVAIL       RAW USED     %RAW USED
      542 TiB     105 TiB      437 TiB         80.67
POOLS:
      NAME                           ID     USED        %USED MAX
AVAIL     OBJECTS
      .rgw.root                      1      1.1 KiB         0 26
TiB            4
      default.rgw.control            2          0 B         0 26
TiB            8
      default.rgw.meta               3       20 MiB         0 26
TiB        75357
      default.rgw.log                4          0 B         0 26
TiB         4271
      default.rgw.buckets.data       5      290 TiB     85.05 51 TiB
78067284
      default.rgw.buckets.non-ec     6          0 B         0 26
TiB            0
      default.rgw.buckets.index      7          0 B         0 26
TiB       603008

- rgw_override_bucket_index_max_shards = 16.   Clients are accessing RGW
via Swift, not S3.

- the replication schema is EC 4+2.

We are using this Ceph cluster as  a secondary storage for another
storage infrastructure (which is more expensive) and we are offloading
cold data (big files with a low number of downloads/reads from our
customer). This way we can lower the TCO .  So most of the files are big
( a few GB at least).

   So far Ceph is doing well considering that I don't have big
expectations from current hardware.  I'm a bit worried however that we
have 78 M objects with max_shards=16 and we will probably reach 100M in
the next few months. Do I need a increase the max shards to ensure the
stability of the cluster ?  I read that storing more than 1 M of objects
in a single bucket can lead to OSD's flapping or having io timeouts
during deep-scrub or even to have ODS's failures due to the leveldb
compacting all the time if we have a large number of DELETEs.

Any advice would be appreciated.


Thank you,

Adrian Nicolae



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux