I think the shard number recommendation is 100K objects/per shard/per bucket. If you have many objects but they are spread in many buckets/containers and each bucket/container have less than 1.6M objects(max_shards=16) then you should be ok. linyunfan Adrian Nicolae <adrian.nicolae@xxxxxxxxxx> 于2020年5月25日周一 下午3:04写道： > > I'm using only Swift , not S3. We have a container for every customer. > Right now there are thousands of containers. > > > > On 5/25/2020 9:02 AM, lin yunfan wrote: > > Can you store your data in different buckets? > > > > linyunfan > > > > Adrian Nicolae <adrian.nicolae@xxxxxxxxxx> 于2020年5月19日周二 下午3:32写道： > >> Hi, > >> > >> I have the following Ceph Mimic setup : > >> > >> - a bunch of old servers with 3-4 SATA drives each (74 OSDs in total) > >> > >> - index/leveldb is stored on each OSD (so no SSD drives, just SATA) > >> > >> - the current usage is : > >> > >> GLOBAL: > >> SIZE AVAIL RAW USED %RAW USED > >> 542 TiB 105 TiB 437 TiB 80.67 > >> POOLS: > >> NAME ID USED %USED MAX > >> AVAIL OBJECTS > >> .rgw.root 1 1.1 KiB 0 26 > >> TiB 4 > >> default.rgw.control 2 0 B 0 26 > >> TiB 8 > >> default.rgw.meta 3 20 MiB 0 26 > >> TiB 75357 > >> default.rgw.log 4 0 B 0 26 > >> TiB 4271 > >> default.rgw.buckets.data 5 290 TiB 85.05 51 TiB > >> 78067284 > >> default.rgw.buckets.non-ec 6 0 B 0 26 > >> TiB 0 > >> default.rgw.buckets.index 7 0 B 0 26 > >> TiB 603008 > >> > >> - rgw_override_bucket_index_max_shards = 16. Clients are accessing RGW > >> via Swift, not S3. > >> > >> - the replication schema is EC 4+2. > >> > >> We are using this Ceph cluster as a secondary storage for another > >> storage infrastructure (which is more expensive) and we are offloading > >> cold data (big files with a low number of downloads/reads from our > >> customer). This way we can lower the TCO . So most of the files are big > >> ( a few GB at least). > >> > >> So far Ceph is doing well considering that I don't have big > >> expectations from current hardware. I'm a bit worried however that we > >> have 78 M objects with max_shards=16 and we will probably reach 100M in > >> the next few months. Do I need a increase the max shards to ensure the > >> stability of the cluster ? I read that storing more than 1 M of objects > >> in a single bucket can lead to OSD's flapping or having io timeouts > >> during deep-scrub or even to have ODS's failures due to the leveldb > >> compacting all the time if we have a large number of DELETEs. > >> > >> Any advice would be appreciated. > >> > >> > >> Thank you, > >> > >> Adrian Nicolae > >> > >> > >> > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |