НА: tcmalloc use a lot of CPU

[Межов Игорь Александрович <megov@xxxxxxxxxx>]

Hi!

We also observe the same behavior on our test Hammer install, and I wrote about it some time ago:

http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/22609

Jan Schremes give us some suggestions in thread, but we still not got any positive results - TCMalloc usage is
high. The usage is lowered to <10%, when disable crc in messages, disable debug and disable cephx auth,
but this is od course not for production use. Also we got a different trace, while performin FIO-RBD benchmarks
on ssd pool:
---
  46,07%  [kernel]              [k] _raw_spin_lock
   6,51%  [kernel]              [k] mb_cache_entry_alloc
   5,74%  libtcmalloc.so.4.2.2  [.] tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)
   5,50%  libtcmalloc.so.4.2.2  [.] tcmalloc::SLL_Next(void*)
   3,86%  libtcmalloc.so.4.2.2  [.] TCMalloc_PageMap3<35>::get(unsigned long) const
   2,73%  libtcmalloc.so.4.2.2  [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)
   0,69%  libtcmalloc.so.4.2.2  [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
   0,69%  libtcmalloc.so.4.2.2  [.] tcmalloc::PageHeap::GetDescriptor(unsigned long) const
   0,64%  libtcmalloc.so.4.2.2  [.] tcmalloc::SLL_PopRange(void**, int, void**, void**)
---

I dont clearly understand, what's happening in this case: ssd pool is connected to the same host,
but different controller (C60X onboard instead of LSI2208), io scheduler set to noop, pool is gathered
from 4х400Gb Intel DC S3700 and have to perform better, I think - more than 30-40 kops.
But we got the trace above and no more then 12-15 kiops. Where can be a problem?






Megov Igor
CIO, Yuterra

---

От: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> от имени YeYin <eyniy@xxxxxx>
Отправлено: 17 августа 2015 г. 12:58
Кому: ceph-users
Тема: tcmalloc use a lot of CPU

 

Hi, all,

  When I do performance test with rados bench, I found tcmalloc consumed a lot of CPU:

Samples: 265K of event 'cycles', Event count (approx.): 104385445900

+  27.58%  libtcmalloc.so.4.1.0    [.] tcmalloc::CentralFreeList::FetchFromSpans()

+  15.25%  libtcmalloc.so.4.1.0    [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long,

+  12.20%  libtcmalloc.so.4.1.0    [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)

+   1.63%  perf                    [.] append_chain

+   1.39%  libtcmalloc.so.4.1.0    [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)

+   1.02%  libtcmalloc.so.4.1.0    [.] tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)

+   0.85%  libtcmalloc.so.4.1.0    [.] 0x0000000000017e6f

+   0.75%  libtcmalloc.so.4.1.0    [.] tcmalloc::ThreadCache::IncreaseCacheLimitLocked()

+   0.67%  libc-2.12.so            [.] memcpy

+   0.53%  libtcmalloc.so.4.1.0    [.] operator delete(void*)

Ceph version:

# ceph --version

ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)

Kernel version:

3.10.83

Is this phenomenon normal?Is there any idea about this problem?

Thanks.

Ye

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com