Hi Wido,
thanks for the tool. Here are some stats from our cluster:
Ceph 12.2.4, 240 OSDs, CephFS only
onodes db_used_bytes avg_obj_size overhead_per_obj
Mean 214871 1574830080 2082298 7607
Max 309855 3018850304 3349799 17753
Min 61390 203423744 285059 3219
STDEV 63324 561838256 726776 2990
See the attached plot as well.
HTH
Dietmar
On 04/26/2018 08:35 PM, Wido den Hollander wrote:
> Hi,
>
> I've been investigating the per object overhead for BlueStore as I've
> seen this has become a topic for a lot of people who want to store a lot
> of small objects in Ceph using BlueStore.
>
> I've writting a piece of Python code which can be run on a server
> running OSDs and will print the overhead.
>
> https://gist.github.com/wido/b1328dd45aae07c45cb8075a24de9f1f
>
> Feedback on this script is welcome, but also the output of what people
> are observing.
>
> The results from my tests are below, but what I see is that the overhead
> seems to range from 10kB to 30kB per object.
>
> On RBD-only clusters the overhead seems to be around 11kB, but on
> clusters with a RGW workload the overhead goes higher to 20kB.
>
> I know that partial overwrites and appends contribute to higher overhead
> on objects and I'm trying to investigate this and share my information
> with the community.
>
> I have two use-cases who want to store >2 billion objects with a avg
> object size of 50kB (8 - 80kB) and the RocksDB overhead is likely to
> become a big problem.
>
> Anybody willing to share the overhead they are seeing with what use-case?
>
> The more data we have on this the better we can estimate how DBs need to
> be sized for BlueStore deployments.
>
> Wido
>
> # Cluster #1
> osd.25 onodes=178572 db_used_bytes=2188378112 avg_obj_size=6196529
> overhead=12254
> osd.20 onodes=209871 db_used_bytes=2307915776 avg_obj_size=5452002
> overhead=10996
> osd.10 onodes=195502 db_used_bytes=2395996160 avg_obj_size=6013645
> overhead=12255
> osd.30 onodes=186172 db_used_bytes=2393899008 avg_obj_size=6359453
> overhead=12858
> osd.1 onodes=169911 db_used_bytes=1799356416 avg_obj_size=4890883
> overhead=10589
> osd.0 onodes=199658 db_used_bytes=2028994560 avg_obj_size=4835928
> overhead=10162
> osd.15 onodes=204015 db_used_bytes=2384461824 avg_obj_size=5722715
> overhead=11687
>
> # Cluster #2
> osd.1 onodes=221735 db_used_bytes=2773483520 avg_obj_size=5742992
> overhead_per_obj=12508
> osd.0 onodes=196817 db_used_bytes=2651848704 avg_obj_size=6454248
> overhead_per_obj=13473
> osd.3 onodes=212401 db_used_bytes=2745171968 avg_obj_size=6004150
> overhead_per_obj=12924
> osd.2 onodes=185757 db_used_bytes=3567255552 avg_obj_size=5359974
> overhead_per_obj=19203
> osd.5 onodes=198822 db_used_bytes=3033530368 avg_obj_size=6765679
> overhead_per_obj=15257
> osd.4 onodes=161142 db_used_bytes=2136997888 avg_obj_size=6377323
> overhead_per_obj=13261
> osd.7 onodes=158951 db_used_bytes=1836056576 avg_obj_size=5247527
> overhead_per_obj=11551
> osd.6 onodes=178874 db_used_bytes=2542796800 avg_obj_size=6539688
> overhead_per_obj=14215
> osd.9 onodes=195166 db_used_bytes=2538602496 avg_obj_size=6237672
> overhead_per_obj=13007
> osd.8 onodes=203946 db_used_bytes=3279945728 avg_obj_size=6523555
> overhead_per_obj=16082
>
> # Cluster 3
> osd.133 onodes=68558 db_used_bytes=15868100608 avg_obj_size=14743206
> overhead_per_obj=231455
> osd.132 onodes=60164 db_used_bytes=13911457792 avg_obj_size=14539445
> overhead_per_obj=231225
> osd.137 onodes=62259 db_used_bytes=15597568000 avg_obj_size=15138484
> overhead_per_obj=250527
> osd.136 onodes=70361 db_used_bytes=14540603392 avg_obj_size=13729154
> overhead_per_obj=206657
> osd.135 onodes=68003 db_used_bytes=12285116416 avg_obj_size=12877744
> overhead_per_obj=180655
> osd.134 onodes=64962 db_used_bytes=14056161280 avg_obj_size=15923550
> overhead_per_obj=216375
> osd.139 onodes=68016 db_used_bytes=20782776320 avg_obj_size=13619345
> overhead_per_obj=305557
> osd.138 onodes=66209 db_used_bytes=12850298880 avg_obj_size=14593418
> overhead_per_obj=194086
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
Attachment:
bluestoreRocksDBoverhead.png
Description: PNG image
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
