Hi, Kefu: we found a mgr instance hang, and attach it with gdb. we can see a signal raised(#8) in or after `DaemonPerfCounters::update(MMgrReport*)` , so its because the tcmalloc heap corrupt. can you help to disgnose the problem? ceph version is 12.0.2. thanks so much. #0 0x00007f2245fc8026 in SpinLock::SlowLock() () from /usr/lib/libtcmalloc.so.4 #1 0x00007f2245fb907a in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /usr/lib/libtcmalloc.so.4 #2 0x00007f2245fbbfaa in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /usr/lib/libtcmalloc.so.4 #3 0x00007f2245fce368 in tc_newarray () from /usr/lib/libtcmalloc.so.4 #4 0x000055840e8f670d in ?? () #5 0x000055840e8f6a9c in pretty_version_to_str[abi:cxx11]() () #6 0x000055840e7122b6 in ceph::BackTrace::print(std::ostream&) const () #7 0x000055840e6f8a27 in ?? () #8 <signal handler called> #9 0x00007f2245fb8c33 in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /usr/lib/libtcmalloc.so.4 #10 0x00007f2245fb8f0a in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /usr/lib/libtcmalloc.so.4 #11 0x00007f2245fb8fbf in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /usr/lib/libtcmalloc.so.4 #12 0x00007f2245fbbfaa in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /usr/lib/libtcmalloc.so.4 #13 0x00007f2245fce368 in tc_newarray () from /usr/lib/libtcmalloc.so.4 #14 0x000055840e67036d in ?? () #15 0x000055840e671e46 in DaemonPerfCounters::update(MMgrReport*) () #16 0x000055840e6770e4 in DaemonServer::handle_report(MMgrReport*) () #17 0x000055840e67cb70 in DaemonServer::ms_dispatch(Message*) () #18 0x000055840e970f6a in DispatchQueue::entry() () #19 0x000055840e809d7d in DispatchQueue::DispatchThread::entry() () #20 0x00007f22453d96ba in start_thread (arg=0x7f223b6dc700) at pthread_create.c:333 #21 0x00007f224444582d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
