Hi,
Here is some stats about OSD memory usage on our lab cluster
(110hdd*1T + 36ssd*200G)
https://drive.google.com/drive/folders/0B1s9jTJ0z59JcmtEX283WUd5bzg?usp=sharing
osd_mem_stats.txt contains stats for hdd and ssd OSDs:
- stats for process memory usage from /proc/pid/status,
- mempools stats
- and heap stats
We use luminous 12.2.1.
We set 9G memory limit in
/lib/systemd/system/ceph-osd@.service.d/override.conf
But that is not enough - OSDs are still been killed because they use
more (with bluestore_cache_size_ssd=3G)
OSDs with hdd (with bluestore_cache_size_hdd=1G) use up to 4.6G.
And, btw, seems that systemd's oom killer looks at VmData, not on VmRss.
Thanks.
On 08/30/2017 06:17 PM, Sage Weil wrote:
Hi Aleksei,
On Wed, 30 Aug 2017, Aleksei Gutikov wrote:
Hi.
I'm trying to synchronize osd daemons memory limits and bluestore cache
settings.
For 12.1.4 we have hdd osds usage about 4G with default settings.
For ssds we have limit 6G and they are been oom killed periodically.
So,
While
osd_op_num_threads_per_shard_hdd=1
osd_op_num_threads_per_shard_ssd=2
and
osd_op_num_shards_hdd=5
osd_op_num_shards_ssd=8
aren't relevant to memory usage. The _per_shard is about how many bytes
are stored in each rocksdb key, and the num_shards is about how many
threads we use.
This is the one that matters:
bluestore_cache_size_hdd=1G
bluestore_cache_size_ssd=3G
It governs how much memory bluestore limits itself to. He bad news is
that bluestore counts what it allocates, not how much memory the allocator
uses, so there is some overhead. From what I've anecdotally seen it's
something like 1.5x, which kind of sucks; there is more to be done here.
On top of that is usage by the OSD outside of bluestore,
which is somewhere in the 500m to 1.5g range.
We're very interested in hearing what observed RSS users see relative to
the configured bluestore size and pg count, along with a dump of the
mempool metadata (ceph daemon osd.NNN dump_mempools).
Does anybody have an idea about the equation for upper bound of memory
consumption?
Very roughly, something like: osd_overhead + bluestore_cache_size * 1.5 ?
Can bluestore_cache_size be decreased safely for example to 2G, or to 1G?
Yes, you can/should change this to whatever you like (big or small).
I want to calculate the maximum expected size of bluestore metadata (that must
be definitely fit into cache) using size of raw space, average size of
objects, rocksdb space amplification.
I thought it should be something simple like
raw_space/avg_obj_size*obj_overhead*rocksdb_space_amp.
For example if obj_overhead=1k, hdd size=1T, rocksdb space amplification is 2
and avg obj size=4M then 1T/4M*1k*2=500M so I need at least 512M for cache.
But wise guys said that I have to take into account number of extents also.
But bluestore_extent_map_shard_max_size=1200, I hope this number is not a
multiplicator...
Nope, just a shard size...
What would be correct approach for calculation of this minimum cache size?
What can be expected size of key-values stored in rocksdb per rados object?
This depends, unfortunately, on what the write pattern is for the objects.
If they're written by RGW in big chunks, the overhead will be smaller.
If it comes out of a 4k random write pattern it will be bigger. Again,
very interestd in hearing user reports of what is observed in real world
situations. I would trust that more than a calculation from first
principles like the above.
Default bluestore_cache_kv_ratio*bluestore_cache_size_ssd=0.99*3G
while default bluestore_cache_kv_max=512M
Looks like BlueStore::_set_cache_sizes() will set cache_kv_ratio to 1/6 in
default case. Is 512M enough for bluestore metadata?
In Mark's testing he found that we got more performance benefit when
small caches were devoted to rocksdb and large caches were devoted mostly
to the bluestore metadata cache (parsed onodes vs caching the encoded
on-disk content). You can always adjust the 512m value upwards (and that
may make sense for large cache sizes). Again, very interested in hearing
whther that works better or worse for your workload!
Thanks-
sage
--
Best regards,
Aleksei Gutikov
Software Engineer | synesis.ru | Minsk. BY
OSD device class: ssd
=====================
heap_dump
---------
+-----------------------------------+----------+-----------+-----------+
| metric | min | mean | max |
+-----------------------------------+----------+-----------+-----------+
| Actual_memory_used_physical_swap | 1.67 Gi | 6.58 Gi | 8.70 Gi |
| Bytes_in_central_cache_freelist | 4.95 Mi | 118.15 Mi | 294.81 Mi |
| Bytes_in_malloc_metadata | 5.01 Mi | 26.56 Mi | 39.82 Mi |
| Bytes_in_page_heap_freelist | 0.0 | 0.0 | 0.0 |
| Bytes_in_thread_cache_freelists | 24.26 Mi | 94.57 Mi | 109.40 Mi |
| Bytes_in_transfer_cache_freelist | 7.04 Mi | 10.93 Mi | 18.05 Mi |
| Bytes_in_use_by_application | 1.63 Gi | 6.34 Gi | 8.40 Gi |
| Bytes_released_to_OS_aka_unmapped | 4.42 Mi | 417.02 Mi | 933.85 Mi |
| Spans_in_use | 44.98 Ki | 383.28 Ki | 611.09 Ki |
| Tcmalloc_page_size | 8.00 Ki | 8.00 Ki | 8.00 Ki |
| Thread_heaps_in_use | 41.0 | 45.8 | 47.0 |
| Virtual_address_space_used | 1.67 Gi | 6.99 Gi | 9.61 Gi |
+-----------------------------------+----------+-----------+-----------+
mempool_bytes
-------------
+----------------------------+----------+-----------+-----------+
| metric | min | mean | max |
+----------------------------+----------+-----------+-----------+
| bloom_filter | 0.0 | 0.0 | 0.0 |
| bluefs | 21.97 Ki | 26.27 Ki | 33.38 Ki |
| bluestore_alloc | 1.07 Mi | 1.38 Mi | 1.72 Mi |
| bluestore_cache_data | 53.20 Mi | 965.17 Mi | 1.73 Gi |
| bluestore_cache_onode | 1.44 Mi | 682.79 Mi | 1.25 Gi |
| bluestore_cache_other | 11.58 Mi | 718.71 Mi | 1.16 Gi |
| bluestore_fsck | 0.0 | 0.0 | 0.0 |
| bluestore_txc | 1.41 Ki | 6.84 Ki | 12.66 Ki |
| bluestore_writing | 87.17 Ki | 693.05 Ki | 905.40 Ki |
| bluestore_writing_deferred | 44.00 Ki | 457.67 Ki | 4.09 Mi |
| buffer_anon | 10.09 Mi | 995.24 Mi | 1.55 Gi |
| buffer_meta | 98.14 Ki | 1.51 Mi | 3.07 Mi |
| mds_co | 0.0 | 0.0 | 0.0 |
| osd | 1.27 Mi | 1.49 Mi | 1.77 Mi |
| osd_mapbl | 0.0 | 338.79 Ki | 1.80 Mi |
| osd_pglog | 68.36 Mi | 89.42 Mi | 117.34 Mi |
| osdmap | 1.11 Mi | 1.26 Mi | 1.38 Mi |
| osdmap_mapping | 0.0 | 0.0 | 0.0 |
| pgmap | 0.0 | 0.0 | 0.0 |
| total | 1.31 Gi | 3.38 Gi | 4.09 Gi |
| unittest_1 | 0.0 | 0.0 | 0.0 |
| unittest_2 | 0.0 | 0.0 | 0.0 |
+----------------------------+----------+-----------+-----------+
mempool_item_size
-----------------
+----------------------------+----------+----------+-----------+
| metric | min | mean | max |
+----------------------------+----------+----------+-----------+
| bloom_filter | 0.0 | 0.0 | 0.0 |
| bluefs | 34.4 | 45.1 | 63.3 |
| bluestore_alloc | 1.0 | 1.0 | 1.0 |
| bluestore_cache_data | 34.73 Ki | 55.22 Ki | 58.67 Ki |
| bluestore_cache_onode | 680.0 | 680.0 | 680.0 |
| bluestore_cache_other | 2.9 | 4.0 | 15.5 |
| bluestore_fsck | 0.0 | 0.0 | 0.0 |
| bluestore_txc | 720.0 | 720.0 | 720.0 |
| bluestore_writing | 43.59 Ki | 65.47 Ki | 69.65 Ki |
| bluestore_writing_deferred | 4.40 Ki | 25.40 Ki | 209.60 Ki |
| buffer_anon | 92.4 | 112.3 | 276.7 |
| buffer_meta | 88.0 | 88.0 | 88.0 |
| mds_co | 0.0 | 0.0 | 0.0 |
| osd | 11.78 Ki | 11.78 Ki | 11.78 Ki |
| osd_mapbl | 0.0 | 14.44 Ki | 47.27 Ki |
| osd_pglog | 217.2 | 261.3 | 279.0 |
| osdmap | 29.2 | 32.4 | 33.1 |
| osdmap_mapping | 0.0 | 0.0 | 0.0 |
| pgmap | 0.0 | 0.0 | 0.0 |
| total | 10.1 | 60.4 | 599.6 |
| unittest_1 | 0.0 | 0.0 | 0.0 |
| unittest_2 | 0.0 | 0.0 | 0.0 |
+----------------------------+----------+----------+-----------+
mempool_items
-------------
+----------------------------+-----------+-----------+-----------+
| metric | min | mean | max |
+----------------------------+-----------+-----------+-----------+
| bloom_filter | 0.0 | 0.0 | 0.0 |
| bluefs | 387.0 | 612.1 | 917.0 |
| bluestore_alloc | 1.07 Mi | 1.38 Mi | 1.72 Mi |
| bluestore_cache_data | 987.0 | 17.46 Ki | 35.55 Ki |
| bluestore_cache_onode | 2.17 Ki | 1.00 Mi | 1.88 Mi |
| bluestore_cache_other | 769.63 Ki | 240.06 Mi | 392.61 Mi |
| bluestore_fsck | 0.0 | 0.0 | 0.0 |
| bluestore_txc | 2.0 | 9.7 | 18.0 |
| bluestore_writing | 2.0 | 10.4 | 13.0 |
| bluestore_writing_deferred | 10.0 | 17.6 | 27.0 |
| buffer_anon | 48.34 Ki | 9.69 Mi | 15.35 Mi |
| buffer_meta | 1.12 Ki | 17.59 Ki | 35.68 Ki |
| mds_co | 0.0 | 0.0 | 0.0 |
| osd | 110.0 | 129.8 | 154.0 |
| osd_mapbl | 0.0 | 7.2 | 39.0 |
| osd_pglog | 267.34 Ki | 353.21 Ki | 501.38 Ki |
| osdmap | 38.36 Ki | 39.74 Ki | 47.61 Ki |
| osdmap_mapping | 0.0 | 0.0 | 0.0 |
| pgmap | 0.0 | 0.0 | 0.0 |
| total | 2.50 Mi | 252.55 Mi | 411.59 Mi |
| unittest_1 | 0.0 | 0.0 | 0.0 |
| unittest_2 | 0.0 | 0.0 | 0.0 |
+----------------------------+-----------+-----------+-----------+
proc_status
-----------
+----------------------------+-----------+-----------+-----------+
| metric | min | mean | max |
+----------------------------+-----------+-----------+-----------+
| RssAnon | 1.65 Gi | 6.56 Gi | 8.66 Gi |
| RssFile | 27.91 Mi | 28.70 Mi | 29.05 Mi |
| RssShmem | 0.0 | 0.0 | 0.0 |
| Threads | 65.0 | 66.9 | 67.0 |
| VmData | 2.30 Gi | 7.63 Gi | 10.26 Gi |
| VmExe | 18.79 Mi | 18.79 Mi | 18.79 Mi |
| VmHWM | 1.68 Gi | 6.77 Gi | 8.92 Gi |
| VmLck | 0.0 | 0.0 | 0.0 |
| VmLib | 15.21 Mi | 15.21 Mi | 15.21 Mi |
| VmPMD | 20.00 Ki | 43.44 Ki | 56.00 Ki |
| VmPTE | 3.87 Mi | 14.52 Mi | 19.77 Mi |
| VmPeak | 2.43 Gi | 8.08 Gi | 10.77 Gi |
| VmPin | 0.0 | 0.0 | 0.0 |
| VmRSS | 1.67 Gi | 6.58 Gi | 8.69 Gi |
| VmSize | 2.43 Gi | 7.77 Gi | 10.40 Gi |
| VmStk | 804.00 Ki | 808.11 Ki | 812.00 Ki |
| VmSwap | 0.0 | 0.0 | 0.0 |
| nonvoluntary_ctxt_switches | 53.0 | 299.6 | 978.0 |
| voluntary_ctxt_switches | 19.93 Ki | 26.59 Ki | 39.16 Ki |
+----------------------------+-----------+-----------+-----------+
OSD device class: hdd
=====================
heap_dump
---------
+-----------------------------------+-----------+-----------+-----------+
| metric | min | mean | max |
+-----------------------------------+-----------+-----------+-----------+
| Actual_memory_used_physical_swap | 2.93 Gi | 3.19 Gi | 3.42 Gi |
| Bytes_in_central_cache_freelist | 173.28 Mi | 231.17 Mi | 285.54 Mi |
| Bytes_in_malloc_metadata | 12.61 Mi | 13.46 Mi | 14.00 Mi |
| Bytes_in_page_heap_freelist | 0.0 | 0.0 | 0.0 |
| Bytes_in_thread_cache_freelists | 74.56 Mi | 93.96 Mi | 105.09 Mi |
| Bytes_in_transfer_cache_freelist | 10.89 Mi | 15.77 Mi | 19.35 Mi |
| Bytes_in_use_by_application | 2.62 Gi | 2.85 Gi | 3.05 Gi |
| Bytes_released_to_OS_aka_unmapped | 95.78 Mi | 285.71 Mi | 480.20 Mi |
| Spans_in_use | 171.07 Ki | 185.28 Ki | 195.04 Ki |
| Tcmalloc_page_size | 8.00 Ki | 8.00 Ki | 8.00 Ki |
| Thread_heaps_in_use | 33.0 | 33.2 | 36.0 |
| Virtual_address_space_used | 3.14 Gi | 3.47 Gi | 3.69 Gi |
+-----------------------------------+-----------+-----------+-----------+
mempool_bytes
-------------
+----------------------------+-----------+-----------+-----------+
| metric | min | mean | max |
+----------------------------+-----------+-----------+-----------+
| bloom_filter | 0.0 | 0.0 | 0.0 |
| bluefs | 9.63 Ki | 11.94 Ki | 13.67 Ki |
| bluestore_alloc | 26.64 Ki | 379.62 Ki | 1.10 Mi |
| bluestore_cache_data | 0.0 | 552.91 Ki | 19.61 Mi |
| bluestore_cache_onode | 187.35 Mi | 192.72 Mi | 197.83 Mi |
| bluestore_cache_other | 304.98 Mi | 318.62 Mi | 322.48 Mi |
| bluestore_fsck | 0.0 | 0.0 | 0.0 |
| bluestore_txc | 1.41 Ki | 53.80 Ki | 113.91 Ki |
| bluestore_writing | 808.52 Ki | 1.25 Mi | 7.77 Mi |
| bluestore_writing_deferred | 6.92 Ki | 730.02 Ki | 4.60 Mi |
| buffer_anon | 835.34 Mi | 967.42 Mi | 1.08 Gi |
| buffer_meta | 7.65 Ki | 10.18 Ki | 20.54 Ki |
| mds_co | 0.0 | 0.0 | 0.0 |
| osd | 1.39 Mi | 1.71 Mi | 1.97 Mi |
| osd_mapbl | 0.0 | 0.0 | 0.0 |
| osd_pglog | 103.92 Mi | 129.63 Mi | 149.63 Mi |
| osdmap | 1.24 Mi | 1.25 Mi | 1.26 Mi |
| osdmap_mapping | 0.0 | 0.0 | 0.0 |
| pgmap | 0.0 | 0.0 | 0.0 |
| total | 1.42 Gi | 1.58 Gi | 1.73 Gi |
| unittest_1 | 0.0 | 0.0 | 0.0 |
| unittest_2 | 0.0 | 0.0 | 0.0 |
+----------------------------+-----------+-----------+-----------+
mempool_item_size
-----------------
+----------------------------+----------+----------+-----------+
| metric | min | mean | max |
+----------------------------+----------+----------+-----------+
| bloom_filter | 0.0 | 0.0 | 0.0 |
| bluefs | 40.0 | 43.1 | 61.8 |
| bluestore_alloc | 1.0 | 1.0 | 1.0 |
| bluestore_cache_data | 0.0 | 15.18 Ki | 248.00 Ki |
| bluestore_cache_onode | 680.0 | 680.0 | 680.0 |
| bluestore_cache_other | 7.1 | 7.3 | 7.4 |
| bluestore_fsck | 0.0 | 0.0 | 0.0 |
| bluestore_txc | 720.0 | 720.0 | 720.0 |
| bluestore_writing | 9.11 Ki | 37.39 Ki | 357.00 Ki |
| bluestore_writing_deferred | 5.16 Ki | 26.33 Ki | 503.09 Ki |
| buffer_anon | 381.1 | 421.3 | 475.5 |
| buffer_meta | 88.0 | 88.0 | 88.0 |
| mds_co | 0.0 | 0.0 | 0.0 |
| osd | 11.78 Ki | 11.78 Ki | 11.78 Ki |
| osd_mapbl | 0.0 | 0.0 | 0.0 |
| osd_pglog | 264.0 | 272.6 | 281.4 |
| osdmap | 32.6 | 32.7 | 33.1 |
| osdmap_mapping | 0.0 | 0.0 | 0.0 |
| pgmap | 0.0 | 0.0 | 0.0 |
| total | 30.9 | 34.3 | 37.6 |
| unittest_1 | 0.0 | 0.0 | 0.0 |
| unittest_2 | 0.0 | 0.0 | 0.0 |
+----------------------------+----------+----------+-----------+
mempool_items
-------------
+----------------------------+-----------+-----------+-----------+
| metric | min | mean | max |
+----------------------------+-----------+-----------+-----------+
| bloom_filter | 0.0 | 0.0 | 0.0 |
| bluefs | 172.0 | 286.0 | 315.0 |
| bluestore_alloc | 26.64 Ki | 379.62 Ki | 1.10 Mi |
| bluestore_cache_data | 0.0 | 3.7 | 147.0 |
| bluestore_cache_onode | 282.12 Ki | 290.21 Ki | 297.91 Ki |
| bluestore_cache_other | 42.30 Mi | 43.65 Mi | 44.03 Mi |
| bluestore_fsck | 0.0 | 0.0 | 0.0 |
| bluestore_txc | 2.0 | 76.5 | 162.0 |
| bluestore_writing | 12.0 | 85.5 | 133.0 |
| bluestore_writing_deferred | 1.0 | 42.3 | 80.0 |
| buffer_anon | 2.18 Mi | 2.30 Mi | 2.35 Mi |
| buffer_meta | 89.0 | 118.4 | 239.0 |
| mds_co | 0.0 | 0.0 | 0.0 |
| osd | 121.0 | 148.9 | 171.0 |
| osd_mapbl | 0.0 | 0.0 | 0.0 |
| osd_pglog | 378.11 Ki | 487.10 Ki | 569.86 Ki |
| osdmap | 38.36 Ki | 39.18 Ki | 39.65 Ki |
| osdmap_mapping | 0.0 | 0.0 | 0.0 |
| pgmap | 0.0 | 0.0 | 0.0 |
| total | 45.39 Mi | 47.11 Mi | 48.00 Mi |
| unittest_1 | 0.0 | 0.0 | 0.0 |
| unittest_2 | 0.0 | 0.0 | 0.0 |
+----------------------------+-----------+-----------+-----------+
proc_status
-----------
+----------------------------+-----------+-----------+-----------+
| metric | min | mean | max |
+----------------------------+-----------+-----------+-----------+
| RssAnon | 2.93 Gi | 3.17 Gi | 3.41 Gi |
| RssFile | 28.10 Mi | 28.65 Mi | 29.01 Mi |
| RssShmem | 0.0 | 0.0 | 0.0 |
| Threads | 56.0 | 56.0 | 56.0 |
| VmData | 3.71 Gi | 4.03 Gi | 4.25 Gi |
| VmExe | 18.79 Mi | 18.79 Mi | 18.79 Mi |
| VmHWM | 3.06 Gi | 3.34 Gi | 3.54 Gi |
| VmLck | 0.0 | 0.0 | 0.0 |
| VmLib | 15.21 Mi | 15.21 Mi | 15.21 Mi |
| VmPMD | 24.00 Ki | 28.62 Ki | 32.00 Ki |
| VmPTE | 6.79 Mi | 7.45 Mi | 7.89 Mi |
| VmPeak | 3.99 Gi | 4.35 Gi | 4.60 Gi |
| VmPin | 0.0 | 0.0 | 0.0 |
| VmRSS | 2.96 Gi | 3.20 Gi | 3.44 Gi |
| VmSize | 3.84 Gi | 4.16 Gi | 4.39 Gi |
| VmStk | 804.00 Ki | 808.40 Ki | 812.00 Ki |
| VmSwap | 0.0 | 0.0 | 0.0 |
| nonvoluntary_ctxt_switches | 462.0 | 24.04 Ki | 40.61 Ki |
| voluntary_ctxt_switches | 43.45 Ki | 61.96 Ki | 89.04 Ki |
+----------------------------+-----------+-----------+-----------+
