Hi, I've been investigating the per object overhead for BlueStore as I've seen this has become a topic for a lot of people who want to store a lot of small objects in Ceph using BlueStore. I've writting a piece of Python code which can be run on a server running OSDs and will print the overhead. https://gist.github.com/wido/b1328dd45aae07c45cb8075a24de9f1f Feedback on this script is welcome, but also the output of what people are observing. The results from my tests are below, but what I see is that the overhead seems to range from 10kB to 30kB per object. On RBD-only clusters the overhead seems to be around 11kB, but on clusters with a RGW workload the overhead goes higher to 20kB. I know that partial overwrites and appends contribute to higher overhead on objects and I'm trying to investigate this and share my information with the community. I have two use-cases who want to store >2 billion objects with a avg object size of 50kB (8 - 80kB) and the RocksDB overhead is likely to become a big problem. Anybody willing to share the overhead they are seeing with what use-case? The more data we have on this the better we can estimate how DBs need to be sized for BlueStore deployments. Wido # Cluster #1 osd.25 onodes=178572 db_used_bytes=2188378112 avg_obj_size=6196529 overhead=12254 osd.20 onodes=209871 db_used_bytes=2307915776 avg_obj_size=5452002 overhead=10996 osd.10 onodes=195502 db_used_bytes=2395996160 avg_obj_size=6013645 overhead=12255 osd.30 onodes=186172 db_used_bytes=2393899008 avg_obj_size=6359453 overhead=12858 osd.1 onodes=169911 db_used_bytes=1799356416 avg_obj_size=4890883 overhead=10589 osd.0 onodes=199658 db_used_bytes=2028994560 avg_obj_size=4835928 overhead=10162 osd.15 onodes=204015 db_used_bytes=2384461824 avg_obj_size=5722715 overhead=11687 # Cluster #2 osd.1 onodes=221735 db_used_bytes=2773483520 avg_obj_size=5742992 overhead_per_obj=12508 osd.0 onodes=196817 db_used_bytes=2651848704 avg_obj_size=6454248 overhead_per_obj=13473 osd.3 onodes=212401 db_used_bytes=2745171968 avg_obj_size=6004150 overhead_per_obj=12924 osd.2 onodes=185757 db_used_bytes=3567255552 avg_obj_size=5359974 overhead_per_obj=19203 osd.5 onodes=198822 db_used_bytes=3033530368 avg_obj_size=6765679 overhead_per_obj=15257 osd.4 onodes=161142 db_used_bytes=2136997888 avg_obj_size=6377323 overhead_per_obj=13261 osd.7 onodes=158951 db_used_bytes=1836056576 avg_obj_size=5247527 overhead_per_obj=11551 osd.6 onodes=178874 db_used_bytes=2542796800 avg_obj_size=6539688 overhead_per_obj=14215 osd.9 onodes=195166 db_used_bytes=2538602496 avg_obj_size=6237672 overhead_per_obj=13007 osd.8 onodes=203946 db_used_bytes=3279945728 avg_obj_size=6523555 overhead_per_obj=16082 # Cluster 3 osd.133 onodes=68558 db_used_bytes=15868100608 avg_obj_size=14743206 overhead_per_obj=231455 osd.132 onodes=60164 db_used_bytes=13911457792 avg_obj_size=14539445 overhead_per_obj=231225 osd.137 onodes=62259 db_used_bytes=15597568000 avg_obj_size=15138484 overhead_per_obj=250527 osd.136 onodes=70361 db_used_bytes=14540603392 avg_obj_size=13729154 overhead_per_obj=206657 osd.135 onodes=68003 db_used_bytes=12285116416 avg_obj_size=12877744 overhead_per_obj=180655 osd.134 onodes=64962 db_used_bytes=14056161280 avg_obj_size=15923550 overhead_per_obj=216375 osd.139 onodes=68016 db_used_bytes=20782776320 avg_obj_size=13619345 overhead_per_obj=305557 osd.138 onodes=66209 db_used_bytes=12850298880 avg_obj_size=14593418 overhead_per_obj=194086 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |