Here are the monitor logs, they're from the monitor starts, however I restarted the osd shortly afterwards and left it to run for 5 or so minutes. http://damoxc.net/ceph/mon.node21.log.gz http://damoxc.net/ceph/mon.node22.log.gz http://damoxc.net/ceph/mon.node23.log.gz On 10 April 2012 22:49, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote: > Nothing apparent from the backtrace. I need monitor logs from when > the osd is sending pg_temp requests. Can you restart the osd and post > the osd and all three monitor logs from when you restarted the osd? > You'll have to enable monitor logging on all three. > -Sam > > On Tue, Apr 10, 2012 at 4:34 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote: >> Okay done that now: >> >> (gdb) thread apply all bt >> >> Thread 61 (Thread 0x7fb5cb00c700 (LWP 12771)): >> #0 0x00007fb5cccdf3f1 in sem_timedwait () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00000000006981f7 in CephContextServiceThread::entry (this=0x2b52bc0) >> at common/ceph_context.cc:53 >> #2 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #3 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x0000000000000000 in ?? () >> >> Thread 60 (Thread 0x7fb5ca80b700 (LWP 12774)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000068abf9 in AdminSocket::entry (this=0x2b5c000) >> at common/admin_socket.cc:211 >> #2 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #3 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x0000000000000000 in ?? () >> >> Thread 59 (Thread 0x7fb5c5801700 (LWP 12831)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000067060e in SimpleMessenger::Accepter::entry (this=0x2b496b8) >> at msg/SimpleMessenger.cc:209 >> #2 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #3 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x0000000000000000 in ?? () >> >> Thread 58 (Thread 0x7fb5c6002700 (LWP 12832)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000672aba in Wait (mutex=..., this=0x2b49aa0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::reaper_entry (this=0x2b49680) >> at msg/SimpleMessenger.cc:2336 >> #3 0x000000000052122d in SimpleMessenger::ReaperThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:522 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 57 (Thread 0x7fb5c6803700 (LWP 12833)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000672aba in Wait (mutex=..., this=0x2b491a0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::reaper_entry (this=0x2b48d80) >> at msg/SimpleMessenger.cc:2336 >> #3 0x000000000052122d in SimpleMessenger::ReaperThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:522 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 56 (Thread 0x7fb5c9008700 (LWP 12834)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000067060e in SimpleMessenger::Accepter::entry (this=0x2b49b38) >> at msg/SimpleMessenger.cc:209 >> #2 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #3 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x0000000000000000 in ?? () >> >> Thread 55 (Thread 0x7fb5ca00a700 (LWP 12835)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000672aba in Wait (mutex=..., this=0x2b49f20) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::reaper_entry (this=0x2b49b00) >> at msg/SimpleMessenger.cc:2336 >> #3 0x000000000052122d in SimpleMessenger::ReaperThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:522 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 54 (Thread 0x7fb5c9809700 (LWP 12836)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000067060e in SimpleMessenger::Accepter::entry (this=0x2b49238) >> at msg/SimpleMessenger.cc:209 >> #2 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #3 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x0000000000000000 in ?? () >> >> Thread 53 (Thread 0x7fb5c8807700 (LWP 12837)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000672aba in Wait (mutex=..., this=0x2b49620) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::reaper_entry (this=0x2b49200) >> at msg/SimpleMessenger.cc:2336 >> #3 0x000000000052122d in SimpleMessenger::ReaperThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:522 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 52 (Thread 0x7fb5c8006700 (LWP 12838)): >> #0 0x00007fb5cb302613 in select () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x00000000006fb210 in SignalHandler::entry (this=0x2b4e420) >> at global/signal_handler.cc:201 >> #2 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #3 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x0000000000000000 in ?? () >> >> Thread 51 (Thread 0x7fb5c7805700 (LWP 12839)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000685bf7 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b66060) at common/Cond.h:67 >> #2 SafeTimer::timer_thread (this=0x2b66050) at common/Timer.cc:110 >> #3 0x000000000068666d in SafeTimerThread::entry (this=<optimized out>) >> at common/Timer.cc:38 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 50 (Thread 0x7fb5c7004700 (LWP 12840)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000685853 in Wait (mutex=..., this=0x2b67460) at common/Cond.h:48 >> #2 SafeTimer::timer_thread (this=0x2b67450) at common/Timer.cc:108 >> #3 0x000000000068666d in SafeTimerThread::entry (this=<optimized out>) >> at common/Timer.cc:38 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 49 (Thread 0x7fb5c5000700 (LWP 12854)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00000000007992d0 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b63500) at ./common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b63500) at ./common/Cond.h:74 >> #3 FileStore::sync_entry (this=0x2b63000) at os/FileStore.cc:3352 >> #4 0x00000000007a65ed in FileStore::SyncThread::entry (this=<optimized out>) >> at os/FileStore.h:103 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 48 (Thread 0x7fb5c47ff700 (LWP 12859)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x000000000077807c in Wait (mutex=..., this=0x2b6ab10) >> at ./common/Cond.h:48 >> #2 FileJournal::write_thread_entry (this=0x2b6a800) at os/FileJournal.cc:1052 >> >> #3 0x00000000005d4c7d in FileJournal::Writer::entry (this=<optimized out>) >> at ./os/FileJournal.h:249 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 47 (Thread 0x7fb5c3ffe700 (LWP 12860)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x000000000077222f in Wait (mutex=..., this=0x2b6a8c0) >> at ./common/Cond.h:48 >> #2 FileJournal::write_finish_thread_entry (this=0x2b6a800) >> at os/FileJournal.cc:1245 >> #3 0x00000000005d4c5d in FileJournal::WriteFinisher::entry ( >> this=<optimized out>) at ./os/FileJournal.h:259 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 46 (Thread 0x7fb5c37fd700 (LWP 12861)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000761f8e in Wait (mutex=..., this=0x2b630b8) >> at ./common/Cond.h:48 >> #2 Finisher::finisher_thread_entry (this=0x2b63070) at common/Finisher.cc:76 >> #3 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #4 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #5 0x0000000000000000 in ?? () >> >> Thread 45 (Thread 0x7fb5c2ffc700 (LWP 12862)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b63868) at common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b63868) at common/Cond.h:74 >> #3 ThreadPool::worker (this=0x2b63810) at common/WorkQueue.cc:71 >> #4 0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>) >> at ./common/WorkQueue.h:121 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 44 (Thread 0x7fb5c27fb700 (LWP 12863)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b63868) at common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b63868) at common/Cond.h:74 >> #3 ThreadPool::worker (this=0x2b63810) at common/WorkQueue.cc:71 >> #4 0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>) >> at ./common/WorkQueue.h:121 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 43 (Thread 0x7fb5c1ffa700 (LWP 12864)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000781ebd in Wait (mutex=..., this=0x2b63970) >> at ./common/Cond.h:48 >> #2 FileStore::flusher_entry (this=0x2b63000) at os/FileStore.cc:3312 >> #3 0x00000000007a49ed in FileStore::FlusherThread::entry ( >> this=<optimized out>) at os/FileStore.h:242 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 42 (Thread 0x7fb5c17f9700 (LWP 12865)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000761f8e in Wait (mutex=..., this=0x2b63750) >> at ./common/Cond.h:48 >> #2 Finisher::finisher_thread_entry (this=0x2b63708) at common/Finisher.cc:76 >> #3 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #4 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #5 0x0000000000000000 in ?? () >> >> Thread 41 (Thread 0x7fb5c0ff8700 (LWP 12866)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000761f8e in Wait (mutex=..., this=0x2b63400) >> at ./common/Cond.h:48 >> #2 Finisher::finisher_thread_entry (this=0x2b633b8) at common/Finisher.cc:76 >> #3 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #4 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #5 0x0000000000000000 in ?? () >> >> Thread 40 (Thread 0x7fb5c07f7700 (LWP 12867)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000685bf7 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b63590) at common/Cond.h:67 >> #2 SafeTimer::timer_thread (this=0x2b63580) at common/Timer.cc:110 >> #3 0x000000000068666d in SafeTimerThread::entry (this=<optimized out>) >> at common/Timer.cc:38 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 39 (Thread 0x7fb5bfff6700 (LWP 12868)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00000000006748da in Wait (mutex=..., this=0x2b49718) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::dispatch_entry (this=0x2b49680) >> at msg/SimpleMessenger.cc:374 >> #3 0x000000000052120d in SimpleMessenger::DispatchThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:559 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 38 (Thread 0x7fb5bf7f5700 (LWP 12869)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00000000006748da in Wait (mutex=..., this=0x2b49298) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::dispatch_entry (this=0x2b49200) >> at msg/SimpleMessenger.cc:374 >> #3 0x000000000052120d in SimpleMessenger::DispatchThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:559 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 37 (Thread 0x7fb5beff4700 (LWP 12870)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00000000006748da in Wait (mutex=..., this=0x2b48e18) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::dispatch_entry (this=0x2b48d80) >> at msg/SimpleMessenger.cc:374 >> #3 0x000000000052120d in SimpleMessenger::DispatchThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:559 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 36 (Thread 0x7fb5be7f3700 (LWP 12871)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00000000006748da in Wait (mutex=..., this=0x2b49b98) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::dispatch_entry (this=0x2b49b00) >> at msg/SimpleMessenger.cc:374 >> #3 0x000000000052120d in SimpleMessenger::DispatchThread::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:559 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 35 (Thread 0x7fb5bdff2700 (LWP 12872)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000685bf7 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x7fff73395748) at common/Cond.h:67 >> #2 SafeTimer::timer_thread (this=0x7fff73395738) at common/Timer.cc:110 >> #3 0x000000000068666d in SafeTimerThread::entry (this=<optimized out>) >> at common/Timer.cc:38 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 34 (Thread 0x7fb5bd7f1700 (LWP 12873)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000761f8e in Wait (mutex=..., this=0x7fff73395838) >> at ./common/Cond.h:48 >> #2 Finisher::finisher_thread_entry (this=0x7fff733957f0) >> at common/Finisher.cc:76 >> #3 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #4 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #5 0x0000000000000000 in ?? () >> >> Thread 33 (Thread 0x7fb5cd2e3700 (LWP 12874)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4965950) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4965780) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 32 (Thread 0x7fb5bcff0700 (LWP 12877)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=31, >> buf=0x7fb5bcfefd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4965780) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 31 (Thread 0x7fb5bceef700 (LWP 12878)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b66470) at common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b66470) at common/Cond.h:74 >> #3 ThreadPool::worker (this=0x2b66418) at common/WorkQueue.cc:71 >> #4 0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>) >> at ./common/WorkQueue.h:121 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 30 (Thread 0x7fb5bc6ee700 (LWP 12879)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b66470) at common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b66470) at common/Cond.h:74 >> #3 ThreadPool::worker (this=0x2b66418) at common/WorkQueue.cc:71 >> #4 0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>) >> at ./common/WorkQueue.h:121 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 29 (Thread 0x7fb5bbeed700 (LWP 12880)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b665a0) at common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b665a0) at common/Cond.h:74 >> #3 ThreadPool::worker (this=0x2b66548) at common/WorkQueue.cc:71 >> #4 0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>) >> at ./common/WorkQueue.h:121 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 28 (Thread 0x7fb5bb6ec700 (LWP 12881)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b666d0) at common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b666d0) at common/Cond.h:74 >> #3 ThreadPool::worker (this=0x2b66678) at common/WorkQueue.cc:71 >> #4 0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>) >> at ./common/WorkQueue.h:121 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 27 (Thread 0x7fb5baeeb700 (LWP 12882)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b66800) at common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b66800) at common/Cond.h:74 >> #3 ThreadPool::worker (this=0x2b667a8) at common/WorkQueue.cc:71 >> #4 0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>) >> at ./common/WorkQueue.h:121 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 26 (Thread 0x7fb5ba6ea700 (LWP 12883)): >> #0 0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x00000000005b43f2 in WaitUntil (when=<optimized out>, mutex=..., >> this=0x2b66918) at ./common/Cond.h:67 >> #2 WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>, >> this=0x2b66918) at ./common/Cond.h:74 >> #3 OSD::heartbeat_entry (this=0x2b66000) at osd/OSD.cc:1699 >> #4 0x00000000005e68ad in OSD::T_Heartbeat::entry (this=<optimized out>) >> at osd/OSD.h:280 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 25 (Thread 0x7fb5b9ee9700 (LWP 12885)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4965e50) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4965c80) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 24 (Thread 0x7fb5b9de8700 (LWP 12886)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4995450) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4995280) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 23 (Thread 0x7fb5b9ce7700 (LWP 12887)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x49951d0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4995000) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 22 (Thread 0x7fb5b9be6700 (LWP 12888)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4995bd0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4995a00) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 21 (Thread 0x7fb5b9ae5700 (LWP 12889)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4995950) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4995780) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 20 (Thread 0x7fb5b99e4700 (LWP 12890)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x49956d0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4995500) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 19 (Thread 0x7fb5b98e3700 (LWP 12891)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4995e50) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4995c80) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 18 (Thread 0x7fb5b97e2700 (LWP 12892)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4d62bd0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4d62a00) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 17 (Thread 0x7fb5b96e1700 (LWP 12893)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=33, >> buf=0x7fb5b96e0d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995280) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 16 (Thread 0x7fb5b95e0700 (LWP 12894)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=35, >> buf=0x7fb5b95dfd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995a00) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 15 (Thread 0x7fb5b94df700 (LWP 12895)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=32, >> buf=0x7fb5b94ded9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4965c80) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 14 (Thread 0x7fb5b93de700 (LWP 12896)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=38, >> buf=0x7fb5b93ddd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995c80) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 13 (Thread 0x7fb5b92dd700 (LWP 12897)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=34, >> buf=0x7fb5b92dcd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995000) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 12 (Thread 0x7fb5b91dc700 (LWP 12898)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=36, >> buf=0x7fb5b91dbd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995780) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 11 (Thread 0x7fb5b90db700 (LWP 12900)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=39, >> buf=0x7fb5b90dad9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4d62a00) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 10 (Thread 0x7fb5b8fda700 (LWP 12899)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=37, >> buf=0x7fb5b8fd9d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995500) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 9 (Thread 0x7fb5b8ed9700 (LWP 12903)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=40, >> buf=0x7fb5b8ed8d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4d62c80) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 8 (Thread 0x7fb5b8dd8700 (LWP 12904)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x4d62e50) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x4d62c80) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 7 (Thread 0x7fb5b8cd7700 (LWP 12908)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=42, >> buf=0x7fb5b8cd6d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x49ed280) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 6 (Thread 0x7fb5b8bd6700 (LWP 12909)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x49ed450) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x49ed280) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 5 (Thread 0x7fb5b8ad5700 (LWP 12910)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=41, >> buf=0x7fb5b8ad4d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x49ed000) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 4 (Thread 0x7fb5b89d4700 (LWP 12911)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x49ed1d0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x49ed000) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 3 (Thread 0x7fb5b88d3700 (LWP 12912)): >> #0 0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x000000000066af87 in tcp_read_wait (sd=<optimized out>, >> timeout=<optimized out>) at msg/tcp.cc:53 >> #2 0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=43, >> buf=0x7fb5b88d2d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26 >> #3 0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x49eda00) >> at msg/SimpleMessenger.cc:1606 >> #4 0x000000000052126d in SimpleMessenger::Pipe::Reader::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:250 >> #5 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #6 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 2 (Thread 0x7fb5b87d2700 (LWP 12913)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000679f1a in Wait (mutex=..., this=0x49edbd0) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::Pipe::writer (this=0x49eda00) >> at msg/SimpleMessenger.cc:1821 >> #3 0x000000000052124d in SimpleMessenger::Pipe::Writer::entry ( >> this=<optimized out>) at msg/SimpleMessenger.h:258 >> #4 0x00007fb5cccd8efc in start_thread () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #5 0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 1 (Thread 0x7fb5cd3017a0 (LWP 12770)): >> #0 0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 () >> from /lib/x86_64-linux-gnu/libpthread.so.0 >> #1 0x0000000000672c47 in Wait (mutex=..., this=0x2b49838) >> at ./common/Cond.h:48 >> #2 SimpleMessenger::wait (this=0x2b49680) at msg/SimpleMessenger.cc:2654 >> #3 0x000000000051efc7 in main (argc=<optimized out>, argv=<optimized out>) >> at ceph_osd.cc:423 >> >> >> On 6 April 2012 22:06, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote: >>> Actually, can you restart osd0, wait the same amount of time, attach >>> to the process with gdb, and post a backtrace from all threads? >>> -Sam >>> >>> On Fri, Apr 6, 2012 at 12:32 PM, Damien Churchill <damoxc@xxxxxxxxx> wrote: >>>> I've upped the monitor debug logging and re-uploaded the osd log, is that okay? >>>> >>>> http://damoxc.net/ceph/mon.node21.log.gz >>>> >>>> On 6 April 2012 19:20, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote: >>>>> Hmm, can you post the monitor logs as well for that period? It looks >>>>> like the osd is requesting a map change and not getting it. >>>>> -Sam >>>>> >>>>> On Fri, Apr 6, 2012 at 10:59 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote: >>>>>> Oops, was set to 600, chmod'd to 644, should be good now. >>>>>> >>>>>> On 6 April 2012 18:56, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote: >>>>>>> Error 403. >>>>>>> -Sam >>>>>>> >>>>>>> On Fri, Apr 6, 2012 at 10:53 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote: >>>>>>>> Okay, uploaded it to http://damoxc.net/ceph/osd.0.log.gz. I restarted >>>>>>>> the osd with debug osd = 20 and let it run for 5 minutes or so. >>>>>>>> >>>>>>>> On 6 April 2012 18:30, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote: >>>>>>>>> Hmm, osd.0 should be enough. I need osd debugging at around 20 from >>>>>>>>> when the osd started. Restarting the osd with debugging at 20 would >>>>>>>>> also work fine. >>>>>>>>> -Sam >>>>>>>>> >>>>>>>>> On Fri, Apr 6, 2012 at 9:55 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote: >>>>>>>>>> I've got that directory on 3 of the osds: 0, 3 and 4. Do you want the >>>>>>>>>> logs to all 3 of them? >>>>>>>>>> >>>>>>>>>> On 6 April 2012 17:50, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote: >>>>>>>>>>> Is there a 0.138_head directory under current/ on any of your osds? >>>>>>>>>>> If so, can you post the log to that osd? I could also use the osd.0 >>>>>>>>>>> log. >>>>>>>>>>> -Sam >>>>>>>>>>> >>>>>>>>>>> On Wed, Apr 4, 2012 at 2:44 PM, Damien Churchill <damoxc@xxxxxxxxx> wrote: >>>>>>>>>>>> I've uploaded them to: >>>>>>>>>>>> >>>>>>>>>>>> http://damoxc.net/ceph/osdmap >>>>>>>>>>>> http://damoxc.net/ceph/pg_dump >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> On 4 April 2012 21:51, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote: >>>>>>>>>>>>> Can you post a copy of your osd map and the output of 'ceph pg dump' ? >>>>>>>>>>>>> You can get the osdmap via 'ceph osd getmap -o <filename>'. >>>>>>>>>>>>> -Sam >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Apr 4, 2012 at 1:12 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm having some trouble getting some pgs to stop being inactive. The >>>>>>>>>>>>>> cluster is running 0.44.1 and the kernel version is 3.2.x. >>>>>>>>>>>>>> >>>>>>>>>>>>>> ceph -s reports: >>>>>>>>>>>>>> 2012-04-04 09:08:57.816029 pg v188540: 990 pgs: 223 inactive, 767 >>>>>>>>>>>>>> active+clean; 205 GB data, 1013 GB used, 8204 GB / 9315 GB avail >>>>>>>>>>>>>> 2012-04-04 09:08:57.817970 mds e2198: 1/1/1 up {0=node24=up:active}, >>>>>>>>>>>>>> 4 up:standby >>>>>>>>>>>>>> 2012-04-04 09:08:57.818024 osd e5910: 5 osds: 5 up, 5 in >>>>>>>>>>>>>> 2012-04-04 09:08:57.818201 log 2012-04-04 09:04:03.838358 osd.3 >>>>>>>>>>>>>> 172.22.10.24:6801/30000 159 : [INF] 0.13d scrub ok >>>>>>>>>>>>>> 2012-04-04 09:08:57.818280 mon e7: 3 mons at >>>>>>>>>>>>>> {node21=172.22.10.21:6789/0,node22=172.22.10.22:6789/0,node23=172.22.10.23:6789/0} >>>>>>>>>>>>>> >>>>>>>>>>>>>> ceph health says: >>>>>>>>>>>>>> 2012-04-04 09:09:01.651053 mon <- [health] >>>>>>>>>>>>>> 2012-04-04 09:09:01.666585 mon.1 -> 'HEALTH_WARN 223 pgs stuck >>>>>>>>>>>>>> inactive; 223 pgs stuck unclean' (0) >>>>>>>>>>>>>> >>>>>>>>>>>>>> I was wondering if anyone has any suggestions about how to resolve >>>>>>>>>>>>>> this, or things to look for. I've tried restarted the ceph daemons on >>>>>>>>>>>>>> the various nodes a few times to no-avail. I don't think that there is >>>>>>>>>>>>>> anything wrong with any of the nodes either. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>>> Damien >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>>>>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
