Re: pgs stuck inactive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Here are the monitor logs, they're from the monitor starts, however I
restarted the osd shortly afterwards and left it to run for 5 or so
minutes.

http://damoxc.net/ceph/mon.node21.log.gz
http://damoxc.net/ceph/mon.node22.log.gz
http://damoxc.net/ceph/mon.node23.log.gz

On 10 April 2012 22:49, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote:
> Nothing apparent from the backtrace.  I need monitor logs from when
> the osd is sending pg_temp requests.  Can you restart the osd and post
> the osd and all three monitor logs from when you restarted the osd?
> You'll have to enable monitor logging on all three.
> -Sam
>
> On Tue, Apr 10, 2012 at 4:34 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote:
>> Okay done that now:
>>
>> (gdb) thread apply all bt
>>
>> Thread 61 (Thread 0x7fb5cb00c700 (LWP 12771)):
>> #0  0x00007fb5cccdf3f1 in sem_timedwait ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00000000006981f7 in CephContextServiceThread::entry (this=0x2b52bc0)
>>    at common/ceph_context.cc:53
>> #2  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #3  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x0000000000000000 in ?? ()
>>
>> Thread 60 (Thread 0x7fb5ca80b700 (LWP 12774)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000068abf9 in AdminSocket::entry (this=0x2b5c000)
>>    at common/admin_socket.cc:211
>> #2  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #3  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x0000000000000000 in ?? ()
>>
>> Thread 59 (Thread 0x7fb5c5801700 (LWP 12831)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000067060e in SimpleMessenger::Accepter::entry (this=0x2b496b8)
>>    at msg/SimpleMessenger.cc:209
>> #2  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #3  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x0000000000000000 in ?? ()
>>
>> Thread 58 (Thread 0x7fb5c6002700 (LWP 12832)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000672aba in Wait (mutex=..., this=0x2b49aa0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::reaper_entry (this=0x2b49680)
>>    at msg/SimpleMessenger.cc:2336
>> #3  0x000000000052122d in SimpleMessenger::ReaperThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:522
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 57 (Thread 0x7fb5c6803700 (LWP 12833)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000672aba in Wait (mutex=..., this=0x2b491a0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::reaper_entry (this=0x2b48d80)
>>    at msg/SimpleMessenger.cc:2336
>> #3  0x000000000052122d in SimpleMessenger::ReaperThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:522
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 56 (Thread 0x7fb5c9008700 (LWP 12834)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000067060e in SimpleMessenger::Accepter::entry (this=0x2b49b38)
>>    at msg/SimpleMessenger.cc:209
>> #2  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #3  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x0000000000000000 in ?? ()
>>
>> Thread 55 (Thread 0x7fb5ca00a700 (LWP 12835)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000672aba in Wait (mutex=..., this=0x2b49f20)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::reaper_entry (this=0x2b49b00)
>>    at msg/SimpleMessenger.cc:2336
>> #3  0x000000000052122d in SimpleMessenger::ReaperThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:522
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 54 (Thread 0x7fb5c9809700 (LWP 12836)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000067060e in SimpleMessenger::Accepter::entry (this=0x2b49238)
>>    at msg/SimpleMessenger.cc:209
>> #2  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #3  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x0000000000000000 in ?? ()
>>
>> Thread 53 (Thread 0x7fb5c8807700 (LWP 12837)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000672aba in Wait (mutex=..., this=0x2b49620)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::reaper_entry (this=0x2b49200)
>>    at msg/SimpleMessenger.cc:2336
>> #3  0x000000000052122d in SimpleMessenger::ReaperThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:522
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 52 (Thread 0x7fb5c8006700 (LWP 12838)):
>> #0  0x00007fb5cb302613 in select () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x00000000006fb210 in SignalHandler::entry (this=0x2b4e420)
>>    at global/signal_handler.cc:201
>> #2  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #3  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x0000000000000000 in ?? ()
>>
>> Thread 51 (Thread 0x7fb5c7805700 (LWP 12839)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000685bf7 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b66060) at common/Cond.h:67
>> #2  SafeTimer::timer_thread (this=0x2b66050) at common/Timer.cc:110
>> #3  0x000000000068666d in SafeTimerThread::entry (this=<optimized out>)
>>    at common/Timer.cc:38
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 50 (Thread 0x7fb5c7004700 (LWP 12840)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000685853 in Wait (mutex=..., this=0x2b67460) at common/Cond.h:48
>> #2  SafeTimer::timer_thread (this=0x2b67450) at common/Timer.cc:108
>> #3  0x000000000068666d in SafeTimerThread::entry (this=<optimized out>)
>>    at common/Timer.cc:38
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 49 (Thread 0x7fb5c5000700 (LWP 12854)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00000000007992d0 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b63500) at ./common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b63500) at ./common/Cond.h:74
>> #3  FileStore::sync_entry (this=0x2b63000) at os/FileStore.cc:3352
>> #4  0x00000000007a65ed in FileStore::SyncThread::entry (this=<optimized out>)
>>    at os/FileStore.h:103
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 48 (Thread 0x7fb5c47ff700 (LWP 12859)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x000000000077807c in Wait (mutex=..., this=0x2b6ab10)
>>    at ./common/Cond.h:48
>> #2  FileJournal::write_thread_entry (this=0x2b6a800) at os/FileJournal.cc:1052
>>
>> #3  0x00000000005d4c7d in FileJournal::Writer::entry (this=<optimized out>)
>>    at ./os/FileJournal.h:249
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 47 (Thread 0x7fb5c3ffe700 (LWP 12860)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x000000000077222f in Wait (mutex=..., this=0x2b6a8c0)
>>    at ./common/Cond.h:48
>> #2  FileJournal::write_finish_thread_entry (this=0x2b6a800)
>>    at os/FileJournal.cc:1245
>> #3  0x00000000005d4c5d in FileJournal::WriteFinisher::entry (
>>    this=<optimized out>) at ./os/FileJournal.h:259
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 46 (Thread 0x7fb5c37fd700 (LWP 12861)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000761f8e in Wait (mutex=..., this=0x2b630b8)
>>    at ./common/Cond.h:48
>> #2  Finisher::finisher_thread_entry (this=0x2b63070) at common/Finisher.cc:76
>> #3  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #4  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #5  0x0000000000000000 in ?? ()
>>
>> Thread 45 (Thread 0x7fb5c2ffc700 (LWP 12862)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b63868) at common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b63868) at common/Cond.h:74
>> #3  ThreadPool::worker (this=0x2b63810) at common/WorkQueue.cc:71
>> #4  0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>)
>>    at ./common/WorkQueue.h:121
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 44 (Thread 0x7fb5c27fb700 (LWP 12863)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b63868) at common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b63868) at common/Cond.h:74
>> #3  ThreadPool::worker (this=0x2b63810) at common/WorkQueue.cc:71
>> #4  0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>)
>>    at ./common/WorkQueue.h:121
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 43 (Thread 0x7fb5c1ffa700 (LWP 12864)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000781ebd in Wait (mutex=..., this=0x2b63970)
>>    at ./common/Cond.h:48
>> #2  FileStore::flusher_entry (this=0x2b63000) at os/FileStore.cc:3312
>> #3  0x00000000007a49ed in FileStore::FlusherThread::entry (
>>    this=<optimized out>) at os/FileStore.h:242
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 42 (Thread 0x7fb5c17f9700 (LWP 12865)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000761f8e in Wait (mutex=..., this=0x2b63750)
>>    at ./common/Cond.h:48
>> #2  Finisher::finisher_thread_entry (this=0x2b63708) at common/Finisher.cc:76
>> #3  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #4  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #5  0x0000000000000000 in ?? ()
>>
>> Thread 41 (Thread 0x7fb5c0ff8700 (LWP 12866)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000761f8e in Wait (mutex=..., this=0x2b63400)
>>    at ./common/Cond.h:48
>> #2  Finisher::finisher_thread_entry (this=0x2b633b8) at common/Finisher.cc:76
>> #3  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #4  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #5  0x0000000000000000 in ?? ()
>>
>> Thread 40 (Thread 0x7fb5c07f7700 (LWP 12867)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000685bf7 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b63590) at common/Cond.h:67
>> #2  SafeTimer::timer_thread (this=0x2b63580) at common/Timer.cc:110
>> #3  0x000000000068666d in SafeTimerThread::entry (this=<optimized out>)
>>    at common/Timer.cc:38
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 39 (Thread 0x7fb5bfff6700 (LWP 12868)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00000000006748da in Wait (mutex=..., this=0x2b49718)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::dispatch_entry (this=0x2b49680)
>>    at msg/SimpleMessenger.cc:374
>> #3  0x000000000052120d in SimpleMessenger::DispatchThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:559
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 38 (Thread 0x7fb5bf7f5700 (LWP 12869)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00000000006748da in Wait (mutex=..., this=0x2b49298)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::dispatch_entry (this=0x2b49200)
>>    at msg/SimpleMessenger.cc:374
>> #3  0x000000000052120d in SimpleMessenger::DispatchThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:559
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 37 (Thread 0x7fb5beff4700 (LWP 12870)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00000000006748da in Wait (mutex=..., this=0x2b48e18)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::dispatch_entry (this=0x2b48d80)
>>    at msg/SimpleMessenger.cc:374
>> #3  0x000000000052120d in SimpleMessenger::DispatchThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:559
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 36 (Thread 0x7fb5be7f3700 (LWP 12871)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00000000006748da in Wait (mutex=..., this=0x2b49b98)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::dispatch_entry (this=0x2b49b00)
>>    at msg/SimpleMessenger.cc:374
>> #3  0x000000000052120d in SimpleMessenger::DispatchThread::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:559
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 35 (Thread 0x7fb5bdff2700 (LWP 12872)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000685bf7 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x7fff73395748) at common/Cond.h:67
>> #2  SafeTimer::timer_thread (this=0x7fff73395738) at common/Timer.cc:110
>> #3  0x000000000068666d in SafeTimerThread::entry (this=<optimized out>)
>>    at common/Timer.cc:38
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 34 (Thread 0x7fb5bd7f1700 (LWP 12873)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000761f8e in Wait (mutex=..., this=0x7fff73395838)
>>    at ./common/Cond.h:48
>> #2  Finisher::finisher_thread_entry (this=0x7fff733957f0)
>>    at common/Finisher.cc:76
>> #3  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #4  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #5  0x0000000000000000 in ?? ()
>>
>> Thread 33 (Thread 0x7fb5cd2e3700 (LWP 12874)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4965950)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4965780)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 32 (Thread 0x7fb5bcff0700 (LWP 12877)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=31,
>>    buf=0x7fb5bcfefd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4965780)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 31 (Thread 0x7fb5bceef700 (LWP 12878)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b66470) at common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b66470) at common/Cond.h:74
>> #3  ThreadPool::worker (this=0x2b66418) at common/WorkQueue.cc:71
>> #4  0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>)
>>    at ./common/WorkQueue.h:121
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 30 (Thread 0x7fb5bc6ee700 (LWP 12879)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b66470) at common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b66470) at common/Cond.h:74
>> #3  ThreadPool::worker (this=0x2b66418) at common/WorkQueue.cc:71
>> #4  0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>)
>>    at ./common/WorkQueue.h:121
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 29 (Thread 0x7fb5bbeed700 (LWP 12880)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b665a0) at common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b665a0) at common/Cond.h:74
>> #3  ThreadPool::worker (this=0x2b66548) at common/WorkQueue.cc:71
>> #4  0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>)
>>    at ./common/WorkQueue.h:121
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 28 (Thread 0x7fb5bb6ec700 (LWP 12881)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b666d0) at common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b666d0) at common/Cond.h:74
>> #3  ThreadPool::worker (this=0x2b66678) at common/WorkQueue.cc:71
>> #4  0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>)
>>    at ./common/WorkQueue.h:121
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 27 (Thread 0x7fb5baeeb700 (LWP 12882)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000688a05 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b66800) at common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b66800) at common/Cond.h:74
>> #3  ThreadPool::worker (this=0x2b667a8) at common/WorkQueue.cc:71
>> #4  0x00000000005d4cad in ThreadPool::WorkThread::entry (this=<optimized out>)
>>    at ./common/WorkQueue.h:121
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 26 (Thread 0x7fb5ba6ea700 (LWP 12883)):
>> #0  0x00007fb5cccdd3cb in pthread_cond_timedwait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x00000000005b43f2 in WaitUntil (when=<optimized out>, mutex=...,
>>    this=0x2b66918) at ./common/Cond.h:67
>> #2  WaitInterval (interval=<optimized out>, mutex=..., cct=<optimized out>,
>>    this=0x2b66918) at ./common/Cond.h:74
>> #3  OSD::heartbeat_entry (this=0x2b66000) at osd/OSD.cc:1699
>> #4  0x00000000005e68ad in OSD::T_Heartbeat::entry (this=<optimized out>)
>>    at osd/OSD.h:280
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 25 (Thread 0x7fb5b9ee9700 (LWP 12885)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4965e50)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4965c80)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 24 (Thread 0x7fb5b9de8700 (LWP 12886)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4995450)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4995280)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 23 (Thread 0x7fb5b9ce7700 (LWP 12887)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x49951d0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4995000)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 22 (Thread 0x7fb5b9be6700 (LWP 12888)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4995bd0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4995a00)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 21 (Thread 0x7fb5b9ae5700 (LWP 12889)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4995950)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4995780)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 20 (Thread 0x7fb5b99e4700 (LWP 12890)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x49956d0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4995500)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 19 (Thread 0x7fb5b98e3700 (LWP 12891)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4995e50)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4995c80)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 18 (Thread 0x7fb5b97e2700 (LWP 12892)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4d62bd0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4d62a00)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 17 (Thread 0x7fb5b96e1700 (LWP 12893)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=33,
>>    buf=0x7fb5b96e0d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995280)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 16 (Thread 0x7fb5b95e0700 (LWP 12894)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=35,
>>    buf=0x7fb5b95dfd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995a00)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 15 (Thread 0x7fb5b94df700 (LWP 12895)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=32,
>>    buf=0x7fb5b94ded9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4965c80)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 14 (Thread 0x7fb5b93de700 (LWP 12896)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=38,
>>    buf=0x7fb5b93ddd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995c80)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 13 (Thread 0x7fb5b92dd700 (LWP 12897)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=34,
>>    buf=0x7fb5b92dcd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995000)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 12 (Thread 0x7fb5b91dc700 (LWP 12898)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=36,
>>    buf=0x7fb5b91dbd9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995780)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 11 (Thread 0x7fb5b90db700 (LWP 12900)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=39,
>>    buf=0x7fb5b90dad9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4d62a00)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 10 (Thread 0x7fb5b8fda700 (LWP 12899)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=37,
>>    buf=0x7fb5b8fd9d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4995500)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 9 (Thread 0x7fb5b8ed9700 (LWP 12903)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=40,
>>    buf=0x7fb5b8ed8d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x4d62c80)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 8 (Thread 0x7fb5b8dd8700 (LWP 12904)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x4d62e50)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x4d62c80)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 7 (Thread 0x7fb5b8cd7700 (LWP 12908)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=42,
>>    buf=0x7fb5b8cd6d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x49ed280)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 6 (Thread 0x7fb5b8bd6700 (LWP 12909)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x49ed450)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x49ed280)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 5 (Thread 0x7fb5b8ad5700 (LWP 12910)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=41,
>>    buf=0x7fb5b8ad4d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x49ed000)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 4 (Thread 0x7fb5b89d4700 (LWP 12911)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x49ed1d0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x49ed000)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 3 (Thread 0x7fb5b88d3700 (LWP 12912)):
>> #0  0x00007fb5cb2fd473 in poll () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x000000000066af87 in tcp_read_wait (sd=<optimized out>,
>>    timeout=<optimized out>) at msg/tcp.cc:53
>> #2  0x000000000066b238 in tcp_read (cct=0x2b4e000, sd=43,
>>    buf=0x7fb5b88d2d9f "\377", len=1, timeout=900000) at msg/tcp.cc:26
>> #3  0x000000000067e2d3 in SimpleMessenger::Pipe::reader (this=0x49eda00)
>>    at msg/SimpleMessenger.cc:1606
>> #4  0x000000000052126d in SimpleMessenger::Pipe::Reader::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:250
>> #5  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #6  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #7  0x0000000000000000 in ?? ()
>>
>> Thread 2 (Thread 0x7fb5b87d2700 (LWP 12913)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000679f1a in Wait (mutex=..., this=0x49edbd0)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::Pipe::writer (this=0x49eda00)
>>    at msg/SimpleMessenger.cc:1821
>> #3  0x000000000052124d in SimpleMessenger::Pipe::Writer::entry (
>>    this=<optimized out>) at msg/SimpleMessenger.h:258
>> #4  0x00007fb5cccd8efc in start_thread ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #5  0x00007fb5cb30959d in clone () from /lib/x86_64-linux-gnu/libc.so.6
>> #6  0x0000000000000000 in ?? ()
>>
>> Thread 1 (Thread 0x7fb5cd3017a0 (LWP 12770)):
>> #0  0x00007fb5cccdd04c in pthread_cond_wait@@GLIBC_2.3.2 ()
>>   from /lib/x86_64-linux-gnu/libpthread.so.0
>> #1  0x0000000000672c47 in Wait (mutex=..., this=0x2b49838)
>>    at ./common/Cond.h:48
>> #2  SimpleMessenger::wait (this=0x2b49680) at msg/SimpleMessenger.cc:2654
>> #3  0x000000000051efc7 in main (argc=<optimized out>, argv=<optimized out>)
>>    at ceph_osd.cc:423
>>
>>
>> On 6 April 2012 22:06, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote:
>>> Actually, can you restart osd0, wait the same amount of time, attach
>>> to the process with gdb, and post a backtrace from all threads?
>>> -Sam
>>>
>>> On Fri, Apr 6, 2012 at 12:32 PM, Damien Churchill <damoxc@xxxxxxxxx> wrote:
>>>> I've upped the monitor debug logging and re-uploaded the osd log, is that okay?
>>>>
>>>> http://damoxc.net/ceph/mon.node21.log.gz
>>>>
>>>> On 6 April 2012 19:20, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote:
>>>>> Hmm, can you post the monitor logs as well for that period?  It looks
>>>>> like the osd is requesting a map change and not getting it.
>>>>> -Sam
>>>>>
>>>>> On Fri, Apr 6, 2012 at 10:59 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote:
>>>>>> Oops, was set to 600, chmod'd to 644, should be good now.
>>>>>>
>>>>>> On 6 April 2012 18:56, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote:
>>>>>>> Error 403.
>>>>>>> -Sam
>>>>>>>
>>>>>>> On Fri, Apr 6, 2012 at 10:53 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote:
>>>>>>>> Okay, uploaded it to http://damoxc.net/ceph/osd.0.log.gz. I restarted
>>>>>>>> the osd with debug osd = 20 and let it run for 5 minutes or so.
>>>>>>>>
>>>>>>>> On 6 April 2012 18:30, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote:
>>>>>>>>> Hmm, osd.0 should be enough.  I need osd debugging at around 20 from
>>>>>>>>> when the osd started.  Restarting the osd with debugging at 20 would
>>>>>>>>> also work fine.
>>>>>>>>> -Sam
>>>>>>>>>
>>>>>>>>> On Fri, Apr 6, 2012 at 9:55 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote:
>>>>>>>>>> I've got that directory on 3 of the osds: 0, 3 and 4. Do you want the
>>>>>>>>>> logs to all 3 of them?
>>>>>>>>>>
>>>>>>>>>> On 6 April 2012 17:50, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>> Is there a 0.138_head directory under current/ on any of your osds?
>>>>>>>>>>> If so, can you post the log to that osd?  I could also use the osd.0
>>>>>>>>>>> log.
>>>>>>>>>>> -Sam
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 4, 2012 at 2:44 PM, Damien Churchill <damoxc@xxxxxxxxx> wrote:
>>>>>>>>>>>> I've uploaded them to:
>>>>>>>>>>>>
>>>>>>>>>>>> http://damoxc.net/ceph/osdmap
>>>>>>>>>>>> http://damoxc.net/ceph/pg_dump
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> On 4 April 2012 21:51, Samuel Just <sam.just@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>> Can you post a copy of your osd map and the output of 'ceph pg dump' ?
>>>>>>>>>>>>>  You can get the osdmap via 'ceph osd getmap -o <filename>'.
>>>>>>>>>>>>> -Sam
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 4, 2012 at 1:12 AM, Damien Churchill <damoxc@xxxxxxxxx> wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm having some trouble getting some pgs to stop being inactive. The
>>>>>>>>>>>>>> cluster is running 0.44.1 and the kernel version is 3.2.x.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ceph -s reports:
>>>>>>>>>>>>>> 2012-04-04 09:08:57.816029    pg v188540: 990 pgs: 223 inactive, 767
>>>>>>>>>>>>>> active+clean; 205 GB data, 1013 GB used, 8204 GB / 9315 GB avail
>>>>>>>>>>>>>> 2012-04-04 09:08:57.817970   mds e2198: 1/1/1 up {0=node24=up:active},
>>>>>>>>>>>>>> 4 up:standby
>>>>>>>>>>>>>> 2012-04-04 09:08:57.818024   osd e5910: 5 osds: 5 up, 5 in
>>>>>>>>>>>>>> 2012-04-04 09:08:57.818201   log 2012-04-04 09:04:03.838358 osd.3
>>>>>>>>>>>>>> 172.22.10.24:6801/30000 159 : [INF] 0.13d scrub ok
>>>>>>>>>>>>>> 2012-04-04 09:08:57.818280   mon e7: 3 mons at
>>>>>>>>>>>>>> {node21=172.22.10.21:6789/0,node22=172.22.10.22:6789/0,node23=172.22.10.23:6789/0}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ceph health says:
>>>>>>>>>>>>>> 2012-04-04 09:09:01.651053 mon <- [health]
>>>>>>>>>>>>>> 2012-04-04 09:09:01.666585 mon.1 -> 'HEALTH_WARN 223 pgs stuck
>>>>>>>>>>>>>> inactive; 223 pgs stuck unclean' (0)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was wondering if anyone has any suggestions about how to resolve
>>>>>>>>>>>>>> this, or things to look for. I've tried restarted the ceph daemons on
>>>>>>>>>>>>>> the various nodes a few times to no-avail. I don't think that there is
>>>>>>>>>>>>>> anything wrong with any of the nodes either.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>> Damien
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>>>>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux