But, you said you need to remove libcmalloc *not* libtcmalloc... I saw librbd/librados is built with libcmalloc not with libtcmalloc.. So, are you saying to remove libtcmalloc (not libcmalloc) to enable jemalloc ? -----Original Message----- From: Stefan Priebe [mailto:s.priebe@xxxxxxxxxxxx] Sent: Wednesday, August 19, 2015 1:31 PM To: Somnath Roy; Alexandre DERUMIER; Mark Nelson Cc: ceph-devel Subject: Re: Ceph Hackathon: More Memory Allocator Testing Am 19.08.2015 um 22:29 schrieb Somnath Roy: > Hmm...We need to fix that as part of configure/Makefile I guess (?).. > Since we have done this jemalloc integration originally, we can take that ownership unless anybody sees a problem of enabling tcmalloc/jemalloc with librbd/librados. > > << You have to remove libcmalloc out of your build environment to get > this done How do I do that ? I am using Ubuntu and can't afford to remove libc* packages. I always use a chroot to build packages where only a minimal bootstrap + the build deps are installed. googleperftools where libtcmalloc comes from is not Ubuntu "core/minimal". Stefan > > Thanks & Regards > Somnath > > -----Original Message----- > From: Stefan Priebe [mailto:s.priebe@xxxxxxxxxxxx] > Sent: Wednesday, August 19, 2015 1:18 PM > To: Somnath Roy; Alexandre DERUMIER; Mark Nelson > Cc: ceph-devel > Subject: Re: Ceph Hackathon: More Memory Allocator Testing > > > Am 19.08.2015 um 22:16 schrieb Somnath Roy: >> Alexandre, >> I am not able to build librados/librbd by using the following config option. >> >> ./configure –without-tcmalloc –with-jemalloc > > Same issue to me. You have to remove libcmalloc out of your build environment to get this done. > > Stefan > > >> It seems it is building osd/mon/Mds/RGW with jemalloc enabled.. >> >> root@emsnode10:~/ceph-latest/src# ldd ./ceph-osd >> linux-vdso.so.1 => (0x00007ffd0eb43000) >> libjemalloc.so.1 => /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 (0x00007f5f92d70000) >> ....... >> >> root@emsnode10:~/ceph-latest/src/.libs# ldd ./librados.so.2.0.0 >> linux-vdso.so.1 => (0x00007ffed46f2000) >> libboost_thread.so.1.55.0 => /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0 (0x00007ff687887000) >> liblttng-ust.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 (0x00007ff68763d000) >> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff687438000) >> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff68721a000) >> libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so (0x00007ff686ee0000) >> libsmime3.so => /usr/lib/x86_64-linux-gnu/libsmime3.so (0x00007ff686cb3000) >> libnspr4.so => /usr/lib/x86_64-linux-gnu/libnspr4.so (0x00007ff686a76000) >> libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007ff686871000) >> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff686668000) >> libboost_system.so.1.55.0 => /usr/lib/x86_64-linux-gnu/libboost_system.so.1.55.0 (0x00007ff686464000) >> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff686160000) >> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff685e59000) >> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff685a94000) >> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff68587e000) >> liblttng-ust-tracepoint.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 (0x00007ff685663000) >> liburcu-bp.so.1 => /usr/lib/liburcu-bp.so.1 (0x00007ff68545c000) >> liburcu-cds.so.1 => /usr/lib/liburcu-cds.so.1 (0x00007ff685255000) >> /lib64/ld-linux-x86-64.so.2 (0x00007ff68a0f6000) >> libnssutil3.so => /usr/lib/x86_64-linux-gnu/libnssutil3.so (0x00007ff685029000) >> libplc4.so => /usr/lib/x86_64-linux-gnu/libplc4.so (0x00007ff684e24000) >> libplds4.so => /usr/lib/x86_64-linux-gnu/libplds4.so >> (0x00007ff684c20000) >> >> It is building with libcmalloc always... >> >> Did you change the ceph makefiles to build librbd/librados with jemalloc ? >> >> Thanks & Regards >> Somnath >> >> -----Original Message----- >> From: ceph-devel-owner@xxxxxxxxxxxxxxx >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Alexandre >> DERUMIER >> Sent: Wednesday, August 19, 2015 7:01 AM >> To: Mark Nelson >> Cc: ceph-devel >> Subject: Re: Ceph Hackathon: More Memory Allocator Testing >> >> Thanks Marc, >> >> Results are matching exactly what I have seen with tcmalloc 2.1 vs 2.4 vs jemalloc. >> >> and indeed tcmalloc, even with bigger cache, seem decrease over time. >> >> >> What is funny, is that I see exactly same behaviour client librbd side, with qemu and multiple iothreads. >> >> >> Switching both server and client to jemalloc give me best performance on small read currently. >> >> >> >> >> >> >> ----- Mail original ----- >> De: "Mark Nelson" <mnelson@xxxxxxxxxx> >> À: "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx> >> Envoyé: Mercredi 19 Août 2015 06:45:36 >> Objet: Ceph Hackathon: More Memory Allocator Testing >> >> Hi Everyone, >> >> One of the goals at the Ceph Hackathon last week was to examine how to improve Ceph Small IO performance. Jian Zhang presented findings showing a dramatic improvement in small random IO performance when Ceph is used with jemalloc. His results build upon Sandisk's original findings that the default thread cache values are a major bottleneck in TCMalloc 2.1. To further verify these results, we sat down at the Hackathon and configured the new performance test cluster that Intel generously donated to the Ceph community laboratory to run through a variety of tests with different memory allocator configurations. I've since written the results of those tests up in pdf form for folks who are interested. >> >> The results are located here: >> >> http://nhm.ceph.com/hackathon/Ceph_Hackathon_Memory_Allocator_Testing. >> pdf >> >> I want to be clear that many other folks have done the heavy lifting here. These results are simply a validation of the many tests that other folks have already done. Many thanks to Sandisk and others for figuring this out as it's a pretty big deal! >> >> Side note: Very little tuning other than swapping the memory allocator and a couple of quick and dirty ceph tunables were set during these tests. It's quite possible that higher IOPS will be achieved as we really start digging into the cluster and learning what the bottlenecks are. >> >> Thanks, >> Mark >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> ________________________________ >> >> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). >> >> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w > j:+v w j m zZ+ ݢj" !tml= >> > N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j:+v w j m zZ+ ݢj" !tml= > ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
