Re: Bluestore with so many small files

[Igor Fedotov <ifedotov@xxxxxxx>]

Hi Behnam,

On 2/12/2018 4:06 PM, Behnam Loghmani wrote:

Hi there,

I am using ceph Luminous 12.2.2 with:

3 osds (each osd is 100G) - no WAL/DB separation.

3 mons

1 rgw

cluster size 3

I stored lots of thumbnails with very small size on ceph with radosgw.

Actual size of files is something about 32G but it filled 70G of each osd.

what's the reason of this high disk usage?

Most probably the major reason is BlueStore allocation granularity. E.g. an object of 1K bytes length needs 64K of disk space if default bluestore_min_alloc_size_hdd  (=64K) is applied.
Additional inconsistency in space reporting might also appear since BlueStore adds up DB volume space when accounting total store space. While free space is taken from Block device only. is As a result when reporting "Used" space always contain that total DB space part ( i.e. Used = Total(Block+DB) - Free(Block) ). That correlates to other comments in this thread about RockDB space usage.
There is a pending PR to fix that:
https://github.com/ceph/ceph/pull/19454/commits/144fb9663778f833782bdcb16acd707c3ed62a86
You may look for "Bluestore: inaccurate disk usage statistics problem" in this mail list for previous discussion as well.

should I change "bluestore_min_alloc_size_hdd"? and If I change it and set it to smaller size, does it impact on performance?

Unfortunately I haven't benchmark "small writes over hdd" cases much hence don't have exacts answer here. Indeed these 'min_alloc_size' family of parameters might impact the performance quite significantly.

what is the best practice for storing small files on bluestore?

Best regards,

Behnam Loghmani

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com