Hmm, I think it's a good perf topic to discuss about buffer alloc/dealloc. For example, maybe frequency alloced object can use memory pool(each pool stores the same objects), but the most challenge to this is also STL structures. On Fri, Nov 14, 2014 at 1:05 AM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote: > On 11/13/2014 10:29 AM, Sage Weil wrote: >> >> On Thu, 13 Nov 2014, Alexandre DERUMIER wrote: >>>>> >>>>> I think we need to figure out why so much time is being spent >>>>> mallocing/freeing memory. Got to get those symbols resolved! >>> >>> >>> Ok, I don't known why, but if I remove all ceph -dbg packages, I'm seeing >>> the rbd && rados symbols now... >>> >>> I have udpdate the files: >>> >>> http://odisoweb1.odiso.net/cephperf/perf-librbd/report.txt >> >> >> Ran it through c++filt: >> >> https://gist.github.com/88ba9409f5d201b957a1 >> >> I'm a bit suprised by the some of the items near the top >> (bufferlist.clear() callers). I'm sure several of those can be >> streamlined to avoid temporary bufferlists. I don't see any super >> egregious users of the allocator, though. >> >> The memcpy callers might be a good place to start... >> >> sage > > > Wasn't josh looking into some of this a year ago? Did anything ever come of > that work? > > >> >> >> >> >> >>> >>> >>> >>> >>> ----- Mail original ----- >>> >>> De: "Mark Nelson" <mark.nelson@xxxxxxxxxxx> >>> ?: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>, "Ceph Devel" >>> <ceph-devel@xxxxxxxxxxxxxxx> >>> Cc: "Mark Nelson" <mark.nelson@xxxxxxxxxxx>, "Sage Weil" >>> <sweil@xxxxxxxxxx>, "Somnath Roy" <somnath.roy@xxxxxxxxxxx> >>> Envoy?: Jeudi 13 Novembre 2014 15:20:40 >>> Objet: Re: client cpu usage : kbrd vs librbd perf report >>> >>> On 11/13/2014 05:15 AM, Alexandre DERUMIER wrote: >>>> >>>> Hi, >>>> >>>> I have redone perf with dwarf >>>> >>>> perf record -g --call-graph dwarf -a -F 99 -- sleep 60 >>>> >>>> I have put perf reports, ceph conf, fio config here: >>>> >>>> http://odisoweb1.odiso.net/cephperf/ >>>> >>>> test setup >>>> ----------- >>>> client cpu config : 8 x Intel(R) Xeon(R) CPU E5-2603 v2 @ 1.80GHz >>>> ceph cluster : 3 nodes (same cpu than client) with 2 osd each (intel ssd >>>> s3500), test pool with replication x1 >>>> rbd volume size : 10G (almost all reads are done in osd buffer cache) >>>> >>>> benchmark with fio 4k randread, with 1 rbd volume. (also tested with 20 >>>> rbd volumes, results are equals). >>>> debian wheezy - kernel 3.17 - and ceph packages from master on >>>> gitbuilder >>>> >>>> (BTW, I have installed librbd/rados dbg packages but I have missing >>>> symbols ?) >>> >>> >>> I think if you run perf report with verbose enabled it will tell you >>> which symbols are missing: >>> >>> perf report -v 2>&1 | less >>> >>> If you have them but it's not detecting them properly you can clean out >>> the cache or even manually reassign the symbols but it's annoying. >>> >>>> >>>> >>>> >>>> Global results: >>>> --------------- >>>> librbd : 60000iops : 98% cpu >>>> krbd : 90000iops : 32% cpu >>>> >>>> >>>> So, librbd usage is 4,5x more than krbd for same ios throughput >>>> >>>> The difference seem to be quite huge, is it expected ? >>> >>> >>> This is kind of the wild west. With that many IOPS we are running into >>> new bottlenecks. :) >>> >>>> >>>> >>>> >>>> >>>> librbd perf report: >>>> ------------------------- >>>> top cpu usage >>>> -------------- >>>> 25.71% fio libc-2.13.so >>>> 17.69% fio librados.so.2.0.0 >>>> 12.38% fio librbd.so.1.0.0 >>>> 27.99% fio [kernel.kallsyms] >>>> 4.19% fio libpthread-2.13.so >>>> >>>> >>>> libc-2.13.so (seem that malloc/free use a lot of cpu here) >>>> ------------ >>>> 21.05%-- _int_malloc >>>> 14.36%-- free >>>> 13.66%-- malloc >>>> 9.89%-- __lll_unlock_wake_private >>>> 5.35%-- __clone >>>> 4.38%-- __poll >>>> 3.77%-- __memcpy_ssse3 >>>> 1.64%-- vfprintf >>>> 1.02%-- arena_get2 >>>> >>> >>> I think we need to figure out why so much time is being spent >>> mallocing/freeing memory. Got to get those symbols resolved! >>> >>>> fio [kernel.kallsyms] : seem to have a lot of futex functions here >>>> ----------------------- >>>> 5.27%-- _raw_spin_lock >>>> 3.88%-- futex_wake >>>> 2.88%-- __switch_to >>>> 2.74%-- system_call >>>> 2.70%-- __schedule >>>> 2.52%-- tcp_sendmsg >>>> 2.47%-- futex_wait_setup >>>> 2.28%-- _raw_spin_lock_irqsave >>>> 2.16%-- idle_cpu >>>> 1.66%-- enqueue_task_fair >>>> 1.57%-- native_write_msr_safe >>>> 1.49%-- hash_futex >>>> 1.46%-- futex_wait >>>> 1.40%-- reschedule_interrupt >>>> 1.37%-- try_to_wake_up >>>> 1.28%-- account_entity_enqueue >>>> 1.25%-- copy_user_enhanced_fast_string >>>> 1.25%-- futex_requeue >>>> 1.24%-- __fget >>>> 1.24%-- update_curr >>>> 1.20%-- tcp_write_xmit >>>> 1.14%-- wake_futex >>>> 1.08%-- scheduler_ipi >>>> 1.05%-- select_task_rq_fair >>>> 1.01%-- dequeue_task_fair >>>> 0.97%-- do_futex >>>> 0.97%-- futex_wait_queue_me >>>> 0.83%-- cpuacct_charge >>>> 0.82%-- tcp_transmit_skb >>>> ... >>>> >>>> >>>> Regards, >>>> >>>> Alexandre >>>> >>>> >>>> >>>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
