On Sun, Sep 9, 2018 at 4:10 PM Xiangyang Yu <penglaiyxy@xxxxxxxxx> wrote: > > Since in jewel 10.2.11，the default filestore omap db is rocksdb，it's > not encouraged to compile with jemalloc. > Right? no, it's not encouraged. but if you insist, you might want to cleanup your building directory and try again. to be specific, you need to remove `src/rocksdb/make_config.mk` , because rocksdb detects its dependencies by itself. in your case, i believe rocksdb/build_tools/build_detect_platform failed to find the rocksdb library, and hence didn't pass -DROCKSDB_JEMALLOC to the compiler. > > If it's right, in Jewel the default msg is still simple msg，jemalloc > performs better. So in jewel 10.2.11， > it's better to use leveldb for both monitor and omap db. > Right？ > > Sage Weil <sage@xxxxxxxxxxxx>于2018年9月8日 周六01:02写道： > > > > On Fri, 7 Sep 2018, Willem Jan Withagen wrote: > > > On 07/09/2018 14:28, Sage Weil wrote: > > > > On Fri, 7 Sep 2018, Xiangyang Yu wrote: > > > > > Hi all, > > > > > > > > > > In our production cluster, we use jewel 10.2.10. We use jemalloc to > > > > > allocate memory. > > > > > These days we are trying to add rocksdb support to osd and monitor, > > > > > > > > Be aware that there is a known problem with rocksdb and jemalloc that > > > > causes a crash; see http://tracker.ceph.com/issues/20557 > > > > > > > > That appears to be a different issue than the compilation problem you are > > > > seeing. Assuming you get past that, though, I would expect you to hit the > > > > #20557 bug anyway. > > > > > > > > Starting with luminous we've recommended users stop using jemalloc because > > > > the switch to AsyncMessenger wipes away the benefit users were seeing in > > > > jewel; tcmalloc and jemalloc now perform about the same (when jemalloc > > > > isn't crashing :). > > > > > > 'mmmm, > > > > > > Made me start to wonder why the FreeBSD version had not been bitten by this? > > > Since jemalloc is the default malloc there. > > > But everything is linked against libtcmalloc.so, which could/should prevent > > > that. > > > > > > Is this codepath typical for Bluestore, and not for filestore?? > > > Since that would be the other explanation. > > > > Right.. this only seems to happen with rocksdb, and thus with luminous. > > (In luminous we also switched the mons to default to rocksdb.) > > > > > But then again I submitted some fixes to rocksdb to get the handling of > > > jemalloc being in libc including/linking the right way. So could be that > > > malloc in rocksdb then started using the libc malloc (aka the jemalloc > > > variant) > > > > > > Do we know what the bug in the rocksdb/jemalloc combination is? Or was it > > > solved by going to tcmalloc, without in depth understanding wat was going on? > > > > Right.. I've no idea what the actual problem is. Disabling jemalloc fixes > > it (most users who hit this had an /etc/{default,sysconfig}/ceph entry > > preloading jemalloc) and jemalloc doesn't offer the same performance boost > > that it did on jewel, so we didn't investigate. > > > > s > > > > > > > > Putting it on the list to figure out. > > > > > > --WjW > > > > > > > > > > > sage > > > > > > > > > > > > > I have merged the commit below but failed to compile the code, > > > > > > > > > > https://github.com/ceph/ceph/pull/18010 > > > > > https://github.com/ceph/ceph/pull/18010 > > > > > > > > > > The screen shows : > > > > > src/rocksdb/db/db_impl.cc:401 undefined reference to > > > > > 'malloc_stats_print' > > > > > Make[3] : [ceph_test_keyvaluedb] ERROR 1 > > > > > src/rocksdb/db/db_impl.cc:401 undefined reference to > > > > > 'malloc_stats_print' > > > > > Make[3] : [ceph_osdmap_tool] ERROR 1 > > > > > src/rocksdb/db/db_impl.cc:401 undefined reference to > > > > > 'malloc_stats_print' > > > > > Make[3] : [ceph_kvstore_tool] ERROR 1 > > > > > > > > > > Then I find some related commit which merged in Lumious： > > > > > cmake: should link against ${ALLOC_LIBS} > > > > > https://github.com/ceph/ceph/pull/11978/files > > > > > > > > > > But this commit did not resolve my problem, there errors still exists. > > > > > > > > > > But when I compile 10.2.11 , no errors show, it's very surprising. ALL > > > > > makefile seems the same. > > > > > > > > > > I have spended one day to solve the problem with no outcome. > > > > > > > > > > I must miss some commits, Anyone has some clues? > > > > > > > > > > Best wished, > > > > > brandy > > > > > > > > > > > -- Regards Kefu Chai
