For Ceph release before nautilus to effect osd_memory_target changes need to restart OSD service. I had similar issue in mimic I did the same in my test setup. Before restarting OSD service ensure you set osd nodown and osd noout similar commands to ensure it doesn't trigger OSD down and recovery. On Wed 13 May, 2020, 10:03 PM Mark Nelson, <mnelson@xxxxxxxxxx> wrote: > Coincidentally Adam on our core team just reported this morning that he > saw extremely high bluestore_cache_other memory usage while running > compression performance tests as well. That may indicate we have a > memory leak related to the compression code. I doubt setting the > memory_target to 3GiB will help in the long run as that will just > attempt to compensate by decreasing the other caches until nothing else > can be shrunk. Adam said he's planning to investigate so hopefully we > will know more soon. > > > Mark > > > > On 5/13/20 10:52 AM, Rafał Wądołowski wrote: > > Mark, > > Unfortunetly I closed terminal with mempool. But there was a lot of > > bytes used by bluestore_cache_other. That was the highest value (about > > 85%). The onode cache takes about 10%. PGlog and osdmaps was okey, low > > values. I saw some ideas that maybe compression_mode force in pool can > > make a mess. > > One more thing, we are running stupid allocator. Right now I am > > decrease the osd_memory_target to 3GiB and will wait if ram problem > > occurs. > > > > > > > > Regards, > > > > */Rafał Wądołowski/* > > > > ------------------------------------------------------------------------ > > *From:* Mark Nelson <mnelson@xxxxxxxxxx> > > *Sent:* Wednesday, May 13, 2020 3:30 PM > > *To:* ceph-users@xxxxxxx <ceph-users@xxxxxxx> > > *Subject:* Re: Memory usage of OSD > > On 5/13/20 12:43 AM, RafaĹ‚ WÄ…doĹ‚owski wrote: > > > Hi, > > > I noticed strange situation in one of our clusters. The OSD deamons > > are taking too much RAM. > > > We are running 12.2.12 and have default configuration of > > osd_memory_target (4GiB). > > > Heap dump shows: > > > > > > osd.2969 dumping heap profile now. > > > ------------------------------------------------ > > > MALLOC: 6381526944 ( 6085.9 MiB) Bytes in use by application > > > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist > > > MALLOC: + 173373288 ( 165.3 MiB) Bytes in central cache freelist > > > MALLOC: + 17163520 ( 16.4 MiB) Bytes in transfer cache freelist > > > MALLOC: + 95339512 ( 90.9 MiB) Bytes in thread cache freelists > > > MALLOC: + 28995744 ( 27.7 MiB) Bytes in malloc metadata > > > MALLOC: ------------ > > > MALLOC: = 6696399008 ( 6386.2 MiB) Actual memory used (physical + > > swap) > > > MALLOC: + 218267648 ( 208.2 MiB) Bytes released to OS (aka > unmapped) > > > MALLOC: ------------ > > > MALLOC: = 6914666656 ( 6594.3 MiB) Virtual address space used > > > MALLOC: > > > MALLOC: 408276 Spans in use > > > MALLOC: 75 Thread heaps in use > > > MALLOC: 8192 Tcmalloc page size > > > ------------------------------------------------ > > > Call ReleaseFreeMemory() to release freelist memory to the OS (via > > madvise()). > > > Bytes released to the OS take up virtual address space but no > > physical memory. > > > > > > IMO "Bytes in use by application" should be less than > > osd_memory_target. Am I correct? > > > I checked heap dump with google-pprof and got following results. > > > Total: 149.4 MB > > > 60.5 40.5% 40.5% 60.5 40.5% > > rocksdb::UncompressBlockContentsForCompressionType > > > 34.2 22.9% 63.4% 34.2 22.9% > > ceph::buffer::create_aligned_in_mempool > > > 11.9 7.9% 71.3% 12.1 8.1% > > std::_Rb_tree::_M_emplace_hint_unique > > > 10.7 7.1% 78.5% 71.2 47.7% rocksdb::ReadBlockContents > > > > > > Does it mean that most of RAM is used by rocksdb? > > > > > > It looks like your heap dump is only accounting for 149.4MB of the > > memory so probably not representative across the whole ~6.5G. Instead > > could you try dumping the mempools via "ceph daemon osd.2969 > > dump_mempools"? > > > > > > > > > > How can I take a deeper look into memory usage ? > > > > > > Beyond looking at the mempools, you can see the bluestore cache > > allocation information by either enabling debug bluestore and debug > > priority_cache_manager 5, or potentially looking at the PCM perf > > counters (I'm not sure if those were in 14.2.12 though). Between the > > heap data, mempool data, and priority cache records, it should become > > clearer what's going on. > > > > > > Mark > > > > > > > > > > > > > Regards, > > > > > > RafaĹ‚ WÄ…doĹ‚owski > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |