Thanks, it's clear from that backtrace what's going on. Opened http://tracker.ceph.com/issues/35985 John On Fri, Sep 14, 2018 at 11:33 AM Christian Albrecht <cal@xxxxxxxx> wrote: > > 14. September 2018 11:31, "John Spray" <jspray@xxxxxxxxxx> schrieb: > > > On Thu, Sep 13, 2018 at 7:55 PM Christian Albrecht <cal@xxxxxxxx> wrote: > > > >> Hi all, > >> ... > >> Let me know I have to provide more information on this. > > > > There was very little change in ceph-mgr between 12.2.7 and 12.2.8, so > > this is strange. > > > > You could try: > > - setting "debug mgr = 20" on a daemon that's exhibiting this behaviour > > - if the ceph-mgr process is still running, use gdb's "attach" > > command and then do a "thread apply all bt" to see where all the > > threads are, in case we can see where something is stuck (might need > > to install the debug package to get symbols for meaningful backtraces) > > > > John > > here is the debug log: > > 2018-09-14 12:05:00.415220 7fc31173a7c0 0 set uid:gid to 167:167 (ceph:ceph) > 2018-09-14 12:05:00.415239 7fc31173a7c0 0 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process ceph-mgr, pid 224042 > 2018-09-14 12:05:00.416819 7fc31173a7c0 0 pidfile_write: ignore empty --pid-file > 2018-09-14 12:05:00.445582 7fc31173a7c0 10 mgr tick tick > 2018-09-14 12:05:00.445588 7fc31173a7c0 1 mgr send_beacon standby > 2018-09-14 12:05:00.445668 7fc31173a7c0 10 mgr send_beacon sending beacon as gid 1754636 > 2018-09-14 12:05:00.445878 7fc31173a7c0 4 mgr init Complete. > 2018-09-14 12:05:00.445886 7fc30888d700 4 mgr ms_dispatch standby mgrmap(e 79) v1 > 2018-09-14 12:05:00.445894 7fc30888d700 4 mgr handle_mgr_map received map epoch 79 > 2018-09-14 12:05:00.445896 7fc30888d700 4 mgr handle_mgr_map active in map: 0 active is 1734502 > 2018-09-14 12:05:00.455091 7fc30888d700 1 mgr init Loading python module 'balancer' > 2018-09-14 12:05:00.461250 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr' > 2018-09-14 12:05:00.475901 7fc30888d700 4 mgr load Standby mode not provided by module 'balancer' > 2018-09-14 12:05:00.475911 7fc30888d700 1 mgr init Loading python module 'dashboard' > 2018-09-14 12:05:00.481582 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr' > 2018-09-14 12:05:00.571933 7fc30888d700 4 mgr load Standby mode available in module 'dashboard' > 2018-09-14 12:05:00.571945 7fc30888d700 1 mgr init Loading python module 'prometheus' > 2018-09-14 12:05:00.578518 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr' > 2018-09-14 12:05:00.634071 7fc30888d700 4 mgr load Standby mode available in module 'prometheus' > 2018-09-14 12:05:00.634084 7fc30888d700 1 mgr init Loading python module 'restful' > 2018-09-14 12:05:00.640297 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr' > 2018-09-14 12:05:00.787052 7fc30888d700 4 mgr load Standby mode not provided by module 'restful' > 2018-09-14 12:05:00.787072 7fc30888d700 1 mgr init Loading python module 'status' > 2018-09-14 12:05:00.792849 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr' > 2018-09-14 12:05:00.813538 7fc30888d700 4 mgr load Standby mode not provided by module 'status' > 2018-09-14 12:05:00.813554 7fc30888d700 4 mgr standby_start Starting modules in standby mode > 2018-09-14 12:05:00.813576 7fc30888d700 4 mgr standby_start skipping module 'balancer' because it does not implement a standby mode > 2018-09-14 12:05:00.813578 7fc30888d700 4 mgr standby_start starting module dashboard > 2018-09-14 12:05:00.813639 7fc30888d700 20 mgr Gil Switched to new thread state 0x564475ffe6e0 > 2018-09-14 12:05:00.813641 7fc2ebfdd700 10 mgr entry listing keys > 2018-09-14 12:05:00.813761 7fc30888d700 1 mgr load Constructed class from module: dashboard > 2018-09-14 12:05:00.813765 7fc30888d700 20 mgr ~Gil Destroying new thread state 0x564475ffe6e0 > 2018-09-14 12:05:00.813770 7fc30888d700 4 mgr start_one Starting thread for dashboard > 2018-09-14 12:05:00.813798 7fc30888d700 4 mgr standby_start starting module prometheus > 2018-09-14 12:05:00.813807 7fc30888d700 20 mgr Gil Switched to new thread state 0x564475ffe6e0 > 2018-09-14 12:05:00.813810 7fc2eb7dc700 4 mgr entry Entering thread for dashboard > 2018-09-14 12:05:00.813855 7fc2eb7dc700 20 mgr Gil Switched to new thread state 0x564475ffe4d0 > 2018-09-14 12:05:00.813886 7fc2eb7dc700 4 mgr get_config get_configkey: mgr/dashboard/ceph06/server_addr > > > and the gdb backtrace: > > > Thread 20 (Thread 0x7fc30da25700 (LWP 224044)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a55935b in ceph::logging::Log::entry (this=0x5644753b4780) at /usr/src/debug/ceph-12.2.8/src/log/Log.cc:460 > #2 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #3 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 19 (Thread 0x7fc30c895700 (LWP 224045)): > #0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a6b3a7a in WaitUntil (when=..., mutex=..., this=0x564475330b10) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:64 > #2 WaitInterval (interval=..., mutex=..., this=0x564475330b10) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:73 > #3 CephContextServiceThread::entry (this=0x564475330a90) at /usr/src/debug/ceph-12.2.8/src/common/ceph_context.cc:135 > #4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 18 (Thread 0x7fc30c094700 (LWP 224046)): > #0 0x00007fc30e3aff0d in poll () from /lib64/libc.so.6 > #1 0x000056446a53db73 in poll (__timeout=-1, __nfds=2, __fds=0x7fc30c092d30) at /usr/include/bits/poll2.h:46 > #2 AdminSocket::entry (this=0x56447535a4e0) at /usr/src/debug/ceph-12.2.8/src/common/admin_socket.cc:250 > #3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 17 (Thread 0x7fc30b893700 (LWP 224047)): > #0 0x00007fc30e3bb183 in epoll_wait () from /lib64/libc.so.6 > #1 0x000056446a7d8ab4 in EpollDriver::event_wait (this=0x5644754850e0, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) > at /usr/src/debug/ceph-12.2.8/src/msg/async/EventEpoll.cc:114 > #2 0x000056446a6103ea in EventCenter::process_events (this=this@entry=0x564475337480, timeout_microseconds=<optimized out>, > timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7fc30b891cf0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Event.cc:395 > #3 0x000056446a6131ee in NetworkStack::__lambda4::operator() (__closure=0x5644753161a0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Stack.cc:51 > #4 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6 > #5 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 16 (Thread 0x7fc30b092700 (LWP 224048)): > #0 0x00007fc30e3bb183 in epoll_wait () from /lib64/libc.so.6 > #1 0x000056446a7d8ab4 in EpollDriver::event_wait (this=0x564475485140, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) > at /usr/src/debug/ceph-12.2.8/src/msg/async/EventEpoll.cc:114 > #2 0x000056446a6103ea in EventCenter::process_events (this=this@entry=0x5644754b7e80, timeout_microseconds=<optimized out>, > timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7fc30b090cf0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Event.cc:395 > #3 0x000056446a6131ee in NetworkStack::__lambda4::operator() (__closure=0x5644753161c0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Stack.cc:51 > #4 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6 > #5 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 15 (Thread 0x7fc30a891700 (LWP 224049)): > #0 0x00007fc30e3bb183 in epoll_wait () from /lib64/libc.so.6 > #1 0x000056446a7d8ab4 in EpollDriver::event_wait (this=0x5644754851a0, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) > at /usr/src/debug/ceph-12.2.8/src/msg/async/EventEpoll.cc:114 > #2 0x000056446a6103ea in EventCenter::process_events (this=this@entry=0x564475336c80, timeout_microseconds=<optimized out>, > timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7fc30a88fcf0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Event.cc:395 > #3 0x000056446a6131ee in NetworkStack::__lambda4::operator() (__closure=0x5644753161b0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Stack.cc:51 > #4 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6 > #5 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #6 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 14 (Thread 0x7fc30a090700 (LWP 224050)): > #0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a42371b in __gthread_cond_timedwait (__abs_timeout=0x7fc30a08ee20, __mutex=<optimized out>, __cond=0x7ffd3bf1d0f0) > at /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/gthr-default.h:871 > #2 __wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., > this=0x7ffd3bf1d0f0) at /usr/include/c++/4.8.2/condition_variable:160 > #3 wait_until<ceph::time_detail::mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., > this=0x7ffd3bf1d0f0) at /usr/include/c++/4.8.2/condition_variable:113 > #4 ceph::timer_detail::timer<ceph::time_detail::mono_clock>::timer_thread (this=0x7ffd3bf1d088) > at /usr/src/debug/ceph-12.2.8/src/common/ceph_timer.h:144 > #5 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6 > #6 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #7 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 13 (Thread 0x7fc30988f700 (LWP 224051)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a553441 in Wait (mutex=..., this=0x564475324fb0) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 Finisher::finisher_thread_entry (this=0x564475324f50) at /usr/src/debug/ceph-12.2.8/src/common/Finisher.cc:101 > #3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 12 (Thread 0x7fc30908e700 (LWP 224052)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a553441 in Wait (mutex=..., this=0x7ffd3bf1dbe8) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 Finisher::finisher_thread_entry (this=0x7ffd3bf1db88) at /usr/src/debug/ceph-12.2.8/src/common/Finisher.cc:101 > #3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 11 (Thread 0x7fc30888d700 (LWP 224053)): > #0 0x00007fc30f2e0afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0 > #1 0x00007fc30f2e0b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0 > #2 0x00007fc30f2e0c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0 > #3 0x00007fc311275735 in PyThread_acquire_lock () from /lib64/libpython2.7.so.1.0 > #4 0x00007fc311241296 in PyEval_RestoreThread () from /lib64/libpython2.7.so.1.0 > #5 0x00007fc31127942e in lock_PyThread_acquire_lock () from /lib64/libpython2.7.so.1.0 > #6 0x00007fc311248cf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #7 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #8 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #9 0x00007fc3112486bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #10 0x00007fc3112486bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #11 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #12 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #13 0x00007fc3112486bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #14 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #15 0x00007fc3111d4978 in function_call () from /lib64/libpython2.7.so.1.0 > #16 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #17 0x00007fc3111bea55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 > #18 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #19 0x00007fc311206a87 in slot_tp_init () from /lib64/libpython2.7.so.1.0 > #20 0x00007fc31120579f in type_call () from /lib64/libpython2.7.so.1.0 > #21 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #22 0x00007fc3112418f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0 > #23 0x000056446a400480 in StandbyPyModule::load (this=0x564475c03420) at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:124 > #24 0x000056446a40160f in StandbyPyModules::start_one (this=0x564476345340, module_name="prometheus", pClass=<optimized out>, > pMyThreadState=...) at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:96 > #25 0x000056446a405265 in PyModuleRegistry::standby_start (this=this@entry=0x7ffd3bf1e680, monc=monc@entry=0x7ffd3bf1c8c8) > at /usr/src/debug/ceph-12.2.8/src/mgr/PyModuleRegistry.cc:321 > #26 0x000056446a41a246 in MgrStandby::handle_mgr_map (this=this@entry=0x7ffd3bf1c8b0, mmap=mmap@entry=0x5644755942c0) > at /usr/src/debug/ceph-12.2.8/src/mgr/MgrStandby.cc:361 > #27 0x000056446a41ab04 in MgrStandby::ms_dispatch (this=0x7ffd3bf1c8b0, m=0x5644755942c0) > at /usr/src/debug/ceph-12.2.8/src/mgr/MgrStandby.cc:376 > #28 0x000056446a815cb2 in ms_deliver_dispatch (m=0x5644755942c0, this=0x564475332700) at /usr/src/debug/ceph-12.2.8/src/msg/Messenger.h:668 > #29 DispatchQueue::entry (this=0x564475332858) at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.cc:197 > #30 0x000056446a5ffbed in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.h:101 > #31 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #32 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 10 (Thread 0x7fc30808c700 (LWP 224054)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a81828f in Wait (mutex=..., this=0x564475332a70) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 DispatchQueue::run_local_delivery (this=0x564475332858) at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.cc:113 > #3 0x000056446a5ffc0d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) > at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.h:115 > #4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 9 (Thread 0x7fc30788b700 (LWP 224055)): > #0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a550e1e in WaitUntil (when=..., mutex=..., this=0x7ffd3bf1ca70) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:64 > #2 SafeTimer::timer_thread (this=0x7ffd3bf1ca60) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:110 > #3 0x000056446a5521ad in SafeTimerThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:30 > #4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 8 (Thread 0x7fc30708a700 (LWP 224056)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a553441 in Wait (mutex=..., this=0x7ffd3bf1cb88) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 Finisher::finisher_thread_entry (this=0x7ffd3bf1cb28) at /usr/src/debug/ceph-12.2.8/src/common/Finisher.cc:101 > #3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 7 (Thread 0x7fc306889700 (LWP 224057)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a550b7f in Wait (mutex=..., this=0x7ffd3bf1d558) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 SafeTimer::timer_thread (this=0x7ffd3bf1d548) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:108 > #3 0x000056446a5521ad in SafeTimerThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:30 > #4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 6 (Thread 0x7fc306088700 (LWP 224058)): > #0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a88c67b in WaitInterval<std::chrono::duration<long> > (interval=..., mutex=..., this=0x564475324ed8) > at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:84 > #2 ObjectCacher::flusher_entry (this=0x564475324d00) at /usr/src/debug/ceph-12.2.8/src/osdc/ObjectCacher.cc:1953 > #3 0x000056446a8987cd in ObjectCacher::FlusherThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/osdc/ObjectCacher.h:432 > #4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #5 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 5 (Thread 0x7fc305887700 (LWP 224059)): > #0 0x00007fc30f2e151d in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x00007fc30f2dce1b in _L_lock_812 () from /lib64/libpthread.so.0 > #2 0x00007fc30f2dcce8 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x000056446a535847 in Mutex::Lock (this=0x7ffd3bf1e560, no_lockdep=no_lockdep@entry=false) > at /usr/src/debug/ceph-12.2.8/src/common/Mutex.cc:107 > #4 0x000056446a5506b2 in SafeTimer::timer_thread (this=0x7ffd3bf1e5b8) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:78 > #5 0x000056446a5521ad in SafeTimerThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:30 > #6 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #7 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 4 (Thread 0x7fc305086700 (LWP 224060)): > #0 0x00007fc30e3aff0d in poll () from /lib64/libc.so.6 > #1 0x000056446a518348 in poll (__timeout=-1, __nfds=4, __fds=0x7fc305084c30) at /usr/include/bits/poll2.h:41 > #2 SignalHandler::entry (this=0x56447535a680) at /usr/src/debug/ceph-12.2.8/src/global/signal_handler.cc:281 > #3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 3 (Thread 0x7fc2ebfdd700 (LWP 224065)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a402c48 in Wait (mutex=..., this=0x7fc2ebfdbc38) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 wait (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:194 > #3 wait (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/mgr/MgrContext.h:39 > #4 JSONCommand::wait (this=this@entry=0x7fc2ebfdbbd0) at /usr/src/debug/ceph-12.2.8/src/mgr/MgrContext.h:53 > #5 0x000056446a4020d1 in StandbyPyModules::LoadConfigThread::entry (this=0x5644763455a0) > at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:141 > #6 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #7 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 2 (Thread 0x7fc2eb7dc700 (LWP 224066)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a402e1e in Cond::Wait (this=0x564476345558, mutex=...) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 0x000056446a400ab5 in with_config<StandbyPyModule::get_config(const string&, std::string*) const::__lambda8> ( > cb=<unknown type in /usr/lib/debug/usr/bin/ceph-mgr.debug, CU 0xc5a7bd, DIE 0xe5b056>, this=0x5644763453d0) > at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.h:76 > #3 StandbyPyModule::get_config (this=0x564475c036c0, key="ceph06/server_addr", value=value@entry=0x7fc2eb7d9ef0) > at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:186 > #4 0x000056446a413a91 in ceph_config_get (self=0x7fc2ec096bd8, args=<optimized out>) > at /usr/src/debug/ceph-12.2.8/src/mgr/BaseMgrStandbyModule.cc:73 > #5 0x00007fc311248cf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #6 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #7 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #8 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #9 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 > #10 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 > #11 0x00007fc3111d4978 in function_call () from /lib64/libpython2.7.so.1.0 > #12 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #13 0x00007fc3111bea55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 > #14 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 > #15 0x00007fc3111afb45 in call_function_tail () from /lib64/libpython2.7.so.1.0 > #16 0x00007fc3111afe7b in PyObject_CallMethod () from /lib64/libpython2.7.so.1.0 > #17 0x000056446a407ffc in PyModuleRunner::serve (this=0x564475c036c0) at /usr/src/debug/ceph-12.2.8/src/mgr/PyModuleRunner.cc:51 > #18 0x000056446a40867f in PyModuleRunner::PyModuleRunnerThread::entry (this=0x564475c036f8) > at /usr/src/debug/ceph-12.2.8/src/mgr/PyModuleRunner.cc:112 > #19 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0 > #20 0x00007fc30e3babad in clone () from /lib64/libc.so.6 > > Thread 1 (Thread 0x7fc31173a7c0 (LWP 224042)): > #0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 > #1 0x000056446a60c863 in Wait (mutex=..., this=0x564475332d78) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48 > #2 AsyncMessenger::wait (this=0x564475332700) at /usr/src/debug/ceph-12.2.8/src/msg/async/AsyncMessenger.cc:481 > #3 0x000056446a36aceb in MgrStandby::main (this=this@entry=0x7ffd3bf1c8b0, args=std::vector of length 0, capacity 0) > at /usr/src/debug/ceph-12.2.8/src/mgr/MgrStandby.cc:435 > #4 0x000056446a369a5a in main (argc=<optimized out>, argv=0x7ffd3bf1e978) at /usr/src/debug/ceph-12.2.8/src/ceph_mgr.cc:74 > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |