14. September 2018 11:31, "John Spray" <jspray@xxxxxxxxxx> schrieb:
> On Thu, Sep 13, 2018 at 7:55 PM Christian Albrecht <cal@xxxxxxxx> wrote:
>
>> Hi all,
>> ...
>> Let me know I have to provide more information on this.
>
> There was very little change in ceph-mgr between 12.2.7 and 12.2.8, so
> this is strange.
>
> You could try:
> - setting "debug mgr = 20" on a daemon that's exhibiting this behaviour
> - if the ceph-mgr process is still running, use gdb's "attach"
> command and then do a "thread apply all bt" to see where all the
> threads are, in case we can see where something is stuck (might need
> to install the debug package to get symbols for meaningful backtraces)
>
> John
here is the debug log:
2018-09-14 12:05:00.415220 7fc31173a7c0 0 set uid:gid to 167:167 (ceph:ceph)
2018-09-14 12:05:00.415239 7fc31173a7c0 0 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable), process ceph-mgr, pid 224042
2018-09-14 12:05:00.416819 7fc31173a7c0 0 pidfile_write: ignore empty --pid-file
2018-09-14 12:05:00.445582 7fc31173a7c0 10 mgr tick tick
2018-09-14 12:05:00.445588 7fc31173a7c0 1 mgr send_beacon standby
2018-09-14 12:05:00.445668 7fc31173a7c0 10 mgr send_beacon sending beacon as gid 1754636
2018-09-14 12:05:00.445878 7fc31173a7c0 4 mgr init Complete.
2018-09-14 12:05:00.445886 7fc30888d700 4 mgr ms_dispatch standby mgrmap(e 79) v1
2018-09-14 12:05:00.445894 7fc30888d700 4 mgr handle_mgr_map received map epoch 79
2018-09-14 12:05:00.445896 7fc30888d700 4 mgr handle_mgr_map active in map: 0 active is 1734502
2018-09-14 12:05:00.455091 7fc30888d700 1 mgr init Loading python module 'balancer'
2018-09-14 12:05:00.461250 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr'
2018-09-14 12:05:00.475901 7fc30888d700 4 mgr load Standby mode not provided by module 'balancer'
2018-09-14 12:05:00.475911 7fc30888d700 1 mgr init Loading python module 'dashboard'
2018-09-14 12:05:00.481582 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr'
2018-09-14 12:05:00.571933 7fc30888d700 4 mgr load Standby mode available in module 'dashboard'
2018-09-14 12:05:00.571945 7fc30888d700 1 mgr init Loading python module 'prometheus'
2018-09-14 12:05:00.578518 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr'
2018-09-14 12:05:00.634071 7fc30888d700 4 mgr load Standby mode available in module 'prometheus'
2018-09-14 12:05:00.634084 7fc30888d700 1 mgr init Loading python module 'restful'
2018-09-14 12:05:00.640297 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr'
2018-09-14 12:05:00.787052 7fc30888d700 4 mgr load Standby mode not provided by module 'restful'
2018-09-14 12:05:00.787072 7fc30888d700 1 mgr init Loading python module 'status'
2018-09-14 12:05:00.792849 7fc30888d700 10 mgr load Computed sys.path '/usr/lib64/python27.zip:/usr/lib64/python2.7/:/usr/lib64/python2.7/plat-linux2:/usr/lib64/python2.7/lib-tk:/usr/lib64/python2.7/lib-old:/usr/lib64/python2.7/lib-dynload:/usr/lib64/python2.7/site-packages:/usr/lib/python2.7/site-packages:/usr/lib/site-python:/usr/lib64/ceph/mgr'
2018-09-14 12:05:00.813538 7fc30888d700 4 mgr load Standby mode not provided by module 'status'
2018-09-14 12:05:00.813554 7fc30888d700 4 mgr standby_start Starting modules in standby mode
2018-09-14 12:05:00.813576 7fc30888d700 4 mgr standby_start skipping module 'balancer' because it does not implement a standby mode
2018-09-14 12:05:00.813578 7fc30888d700 4 mgr standby_start starting module dashboard
2018-09-14 12:05:00.813639 7fc30888d700 20 mgr Gil Switched to new thread state 0x564475ffe6e0
2018-09-14 12:05:00.813641 7fc2ebfdd700 10 mgr entry listing keys
2018-09-14 12:05:00.813761 7fc30888d700 1 mgr load Constructed class from module: dashboard
2018-09-14 12:05:00.813765 7fc30888d700 20 mgr ~Gil Destroying new thread state 0x564475ffe6e0
2018-09-14 12:05:00.813770 7fc30888d700 4 mgr start_one Starting thread for dashboard
2018-09-14 12:05:00.813798 7fc30888d700 4 mgr standby_start starting module prometheus
2018-09-14 12:05:00.813807 7fc30888d700 20 mgr Gil Switched to new thread state 0x564475ffe6e0
2018-09-14 12:05:00.813810 7fc2eb7dc700 4 mgr entry Entering thread for dashboard
2018-09-14 12:05:00.813855 7fc2eb7dc700 20 mgr Gil Switched to new thread state 0x564475ffe4d0
2018-09-14 12:05:00.813886 7fc2eb7dc700 4 mgr get_config get_configkey: mgr/dashboard/ceph06/server_addr
and the gdb backtrace:
Thread 20 (Thread 0x7fc30da25700 (LWP 224044)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a55935b in ceph::logging::Log::entry (this=0x5644753b4780) at /usr/src/debug/ceph-12.2.8/src/log/Log.cc:460
#2 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#3 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fc30c895700 (LWP 224045)):
#0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a6b3a7a in WaitUntil (when=..., mutex=..., this=0x564475330b10) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:64
#2 WaitInterval (interval=..., mutex=..., this=0x564475330b10) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:73
#3 CephContextServiceThread::entry (this=0x564475330a90) at /usr/src/debug/ceph-12.2.8/src/common/ceph_context.cc:135
#4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fc30c094700 (LWP 224046)):
#0 0x00007fc30e3aff0d in poll () from /lib64/libc.so.6
#1 0x000056446a53db73 in poll (__timeout=-1, __nfds=2, __fds=0x7fc30c092d30) at /usr/include/bits/poll2.h:46
#2 AdminSocket::entry (this=0x56447535a4e0) at /usr/src/debug/ceph-12.2.8/src/common/admin_socket.cc:250
#3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fc30b893700 (LWP 224047)):
#0 0x00007fc30e3bb183 in epoll_wait () from /lib64/libc.so.6
#1 0x000056446a7d8ab4 in EpollDriver::event_wait (this=0x5644754850e0, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>)
at /usr/src/debug/ceph-12.2.8/src/msg/async/EventEpoll.cc:114
#2 0x000056446a6103ea in EventCenter::process_events (this=this@entry=0x564475337480, timeout_microseconds=<optimized out>,
timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7fc30b891cf0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Event.cc:395
#3 0x000056446a6131ee in NetworkStack::__lambda4::operator() (__closure=0x5644753161a0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Stack.cc:51
#4 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6
#5 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fc30b092700 (LWP 224048)):
#0 0x00007fc30e3bb183 in epoll_wait () from /lib64/libc.so.6
#1 0x000056446a7d8ab4 in EpollDriver::event_wait (this=0x564475485140, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>)
at /usr/src/debug/ceph-12.2.8/src/msg/async/EventEpoll.cc:114
#2 0x000056446a6103ea in EventCenter::process_events (this=this@entry=0x5644754b7e80, timeout_microseconds=<optimized out>,
timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7fc30b090cf0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Event.cc:395
#3 0x000056446a6131ee in NetworkStack::__lambda4::operator() (__closure=0x5644753161c0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Stack.cc:51
#4 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6
#5 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fc30a891700 (LWP 224049)):
#0 0x00007fc30e3bb183 in epoll_wait () from /lib64/libc.so.6
#1 0x000056446a7d8ab4 in EpollDriver::event_wait (this=0x5644754851a0, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>)
at /usr/src/debug/ceph-12.2.8/src/msg/async/EventEpoll.cc:114
#2 0x000056446a6103ea in EventCenter::process_events (this=this@entry=0x564475336c80, timeout_microseconds=<optimized out>,
timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7fc30a88fcf0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Event.cc:395
#3 0x000056446a6131ee in NetworkStack::__lambda4::operator() (__closure=0x5644753161b0) at /usr/src/debug/ceph-12.2.8/src/msg/async/Stack.cc:51
#4 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6
#5 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fc30a090700 (LWP 224050)):
#0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a42371b in __gthread_cond_timedwait (__abs_timeout=0x7fc30a08ee20, __mutex=<optimized out>, __cond=0x7ffd3bf1d0f0)
at /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/gthr-default.h:871
#2 __wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=...,
this=0x7ffd3bf1d0f0) at /usr/include/c++/4.8.2/condition_variable:160
#3 wait_until<ceph::time_detail::mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=...,
this=0x7ffd3bf1d0f0) at /usr/include/c++/4.8.2/condition_variable:113
#4 ceph::timer_detail::timer<ceph::time_detail::mono_clock>::timer_thread (this=0x7ffd3bf1d088)
at /usr/src/debug/ceph-12.2.8/src/common/ceph_timer.h:144
#5 0x00007fc30ec56070 in ?? () from /lib64/libstdc++.so.6
#6 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fc30988f700 (LWP 224051)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a553441 in Wait (mutex=..., this=0x564475324fb0) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 Finisher::finisher_thread_entry (this=0x564475324f50) at /usr/src/debug/ceph-12.2.8/src/common/Finisher.cc:101
#3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fc30908e700 (LWP 224052)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a553441 in Wait (mutex=..., this=0x7ffd3bf1dbe8) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 Finisher::finisher_thread_entry (this=0x7ffd3bf1db88) at /usr/src/debug/ceph-12.2.8/src/common/Finisher.cc:101
#3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fc30888d700 (LWP 224053)):
#0 0x00007fc30f2e0afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1 0x00007fc30f2e0b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x00007fc30f2e0c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x00007fc311275735 in PyThread_acquire_lock () from /lib64/libpython2.7.so.1.0
#4 0x00007fc311241296 in PyEval_RestoreThread () from /lib64/libpython2.7.so.1.0
#5 0x00007fc31127942e in lock_PyThread_acquire_lock () from /lib64/libpython2.7.so.1.0
#6 0x00007fc311248cf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#7 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#8 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9 0x00007fc3112486bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#10 0x00007fc3112486bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#11 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#12 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#13 0x00007fc3112486bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#14 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#15 0x00007fc3111d4978 in function_call () from /lib64/libpython2.7.so.1.0
#16 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#17 0x00007fc3111bea55 in instancemethod_call () from /lib64/libpython2.7.so.1.0
#18 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#19 0x00007fc311206a87 in slot_tp_init () from /lib64/libpython2.7.so.1.0
#20 0x00007fc31120579f in type_call () from /lib64/libpython2.7.so.1.0
#21 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#22 0x00007fc3112418f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#23 0x000056446a400480 in StandbyPyModule::load (this=0x564475c03420) at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:124
#24 0x000056446a40160f in StandbyPyModules::start_one (this=0x564476345340, module_name="prometheus", pClass=<optimized out>,
pMyThreadState=...) at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:96
#25 0x000056446a405265 in PyModuleRegistry::standby_start (this=this@entry=0x7ffd3bf1e680, monc=monc@entry=0x7ffd3bf1c8c8)
at /usr/src/debug/ceph-12.2.8/src/mgr/PyModuleRegistry.cc:321
#26 0x000056446a41a246 in MgrStandby::handle_mgr_map (this=this@entry=0x7ffd3bf1c8b0, mmap=mmap@entry=0x5644755942c0)
at /usr/src/debug/ceph-12.2.8/src/mgr/MgrStandby.cc:361
#27 0x000056446a41ab04 in MgrStandby::ms_dispatch (this=0x7ffd3bf1c8b0, m=0x5644755942c0)
at /usr/src/debug/ceph-12.2.8/src/mgr/MgrStandby.cc:376
#28 0x000056446a815cb2 in ms_deliver_dispatch (m=0x5644755942c0, this=0x564475332700) at /usr/src/debug/ceph-12.2.8/src/msg/Messenger.h:668
#29 DispatchQueue::entry (this=0x564475332858) at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.cc:197
#30 0x000056446a5ffbed in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.h:101
#31 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#32 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fc30808c700 (LWP 224054)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a81828f in Wait (mutex=..., this=0x564475332a70) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 DispatchQueue::run_local_delivery (this=0x564475332858) at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.cc:113
#3 0x000056446a5ffc0d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>)
at /usr/src/debug/ceph-12.2.8/src/msg/DispatchQueue.h:115
#4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fc30788b700 (LWP 224055)):
#0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a550e1e in WaitUntil (when=..., mutex=..., this=0x7ffd3bf1ca70) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:64
#2 SafeTimer::timer_thread (this=0x7ffd3bf1ca60) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:110
#3 0x000056446a5521ad in SafeTimerThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:30
#4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fc30708a700 (LWP 224056)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a553441 in Wait (mutex=..., this=0x7ffd3bf1cb88) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 Finisher::finisher_thread_entry (this=0x7ffd3bf1cb28) at /usr/src/debug/ceph-12.2.8/src/common/Finisher.cc:101
#3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fc306889700 (LWP 224057)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a550b7f in Wait (mutex=..., this=0x7ffd3bf1d558) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 SafeTimer::timer_thread (this=0x7ffd3bf1d548) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:108
#3 0x000056446a5521ad in SafeTimerThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:30
#4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fc306088700 (LWP 224058)):
#0 0x00007fc30f2ded42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a88c67b in WaitInterval<std::chrono::duration<long> > (interval=..., mutex=..., this=0x564475324ed8)
at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:84
#2 ObjectCacher::flusher_entry (this=0x564475324d00) at /usr/src/debug/ceph-12.2.8/src/osdc/ObjectCacher.cc:1953
#3 0x000056446a8987cd in ObjectCacher::FlusherThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/osdc/ObjectCacher.h:432
#4 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fc305887700 (LWP 224059)):
#0 0x00007fc30f2e151d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fc30f2dce1b in _L_lock_812 () from /lib64/libpthread.so.0
#2 0x00007fc30f2dcce8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000056446a535847 in Mutex::Lock (this=0x7ffd3bf1e560, no_lockdep=no_lockdep@entry=false)
at /usr/src/debug/ceph-12.2.8/src/common/Mutex.cc:107
#4 0x000056446a5506b2 in SafeTimer::timer_thread (this=0x7ffd3bf1e5b8) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:78
#5 0x000056446a5521ad in SafeTimerThread::entry (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Timer.cc:30
#6 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fc305086700 (LWP 224060)):
#0 0x00007fc30e3aff0d in poll () from /lib64/libc.so.6
#1 0x000056446a518348 in poll (__timeout=-1, __nfds=4, __fds=0x7fc305084c30) at /usr/include/bits/poll2.h:41
#2 SignalHandler::entry (this=0x56447535a680) at /usr/src/debug/ceph-12.2.8/src/global/signal_handler.cc:281
#3 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fc2ebfdd700 (LWP 224065)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a402c48 in Wait (mutex=..., this=0x7fc2ebfdbc38) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 wait (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:194
#3 wait (this=<optimized out>) at /usr/src/debug/ceph-12.2.8/src/mgr/MgrContext.h:39
#4 JSONCommand::wait (this=this@entry=0x7fc2ebfdbbd0) at /usr/src/debug/ceph-12.2.8/src/mgr/MgrContext.h:53
#5 0x000056446a4020d1 in StandbyPyModules::LoadConfigThread::entry (this=0x5644763455a0)
at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:141
#6 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fc2eb7dc700 (LWP 224066)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a402e1e in Cond::Wait (this=0x564476345558, mutex=...) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 0x000056446a400ab5 in with_config<StandbyPyModule::get_config(const string&, std::string*) const::__lambda8> (
cb=<unknown type in /usr/lib/debug/usr/bin/ceph-mgr.debug, CU 0xc5a7bd, DIE 0xe5b056>, this=0x5644763453d0)
at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.h:76
#3 StandbyPyModule::get_config (this=0x564475c036c0, key="ceph06/server_addr", value=value@entry=0x7fc2eb7d9ef0)
at /usr/src/debug/ceph-12.2.8/src/mgr/StandbyPyModules.cc:186
#4 0x000056446a413a91 in ceph_config_get (self=0x7fc2ec096bd8, args=<optimized out>)
at /usr/src/debug/ceph-12.2.8/src/mgr/BaseMgrStandbyModule.cc:73
#5 0x00007fc311248cf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#6 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#7 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#8 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#9 0x00007fc31124853c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#10 0x00007fc31124b03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#11 0x00007fc3111d4978 in function_call () from /lib64/libpython2.7.so.1.0
#12 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#13 0x00007fc3111bea55 in instancemethod_call () from /lib64/libpython2.7.so.1.0
#14 0x00007fc3111afa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#15 0x00007fc3111afb45 in call_function_tail () from /lib64/libpython2.7.so.1.0
#16 0x00007fc3111afe7b in PyObject_CallMethod () from /lib64/libpython2.7.so.1.0
#17 0x000056446a407ffc in PyModuleRunner::serve (this=0x564475c036c0) at /usr/src/debug/ceph-12.2.8/src/mgr/PyModuleRunner.cc:51
#18 0x000056446a40867f in PyModuleRunner::PyModuleRunnerThread::entry (this=0x564475c036f8)
at /usr/src/debug/ceph-12.2.8/src/mgr/PyModuleRunner.cc:112
#19 0x00007fc30f2dae25 in start_thread () from /lib64/libpthread.so.0
#20 0x00007fc30e3babad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc31173a7c0 (LWP 224042)):
#0 0x00007fc30f2de995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000056446a60c863 in Wait (mutex=..., this=0x564475332d78) at /usr/src/debug/ceph-12.2.8/src/common/Cond.h:48
#2 AsyncMessenger::wait (this=0x564475332700) at /usr/src/debug/ceph-12.2.8/src/msg/async/AsyncMessenger.cc:481
#3 0x000056446a36aceb in MgrStandby::main (this=this@entry=0x7ffd3bf1c8b0, args=std::vector of length 0, capacity 0)
at /usr/src/debug/ceph-12.2.8/src/mgr/MgrStandby.cc:435
#4 0x000056446a369a5a in main (argc=<optimized out>, argv=0x7ffd3bf1e978) at /usr/src/debug/ceph-12.2.8/src/ceph_mgr.cc:74
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
