Hi, during recovery testing on a latest firefly with leveldb backend we found that the OSDs on a selected host may crash at once, leaving attached backtrace. In other ways, recovery goes more or less smoothly for hours. Timestamps shows how the issue is correlated between different processes on same node: core.ceph-osd.25426.node01.1414148261 core.ceph-osd.25734.node01.1414148263 core.ceph-osd.25566.node01.1414148345 The question is about kv backend state in Firefly - is it considered stable enough to run production test against it or we should better move to giant/master for this? Thanks!
GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/ceph-osd...Reading symbols from /usr/lib/debug/usr/bin/ceph-osd...done. done. [New LWP 10182] [New LWP 10183] [New LWP 10699] [New LWP 10184] [New LWP 10703] [New LWP 10704] [New LWP 10702] [New LWP 10708] [New LWP 10707] [New LWP 10710] [New LWP 10700] [New LWP 10717] [New LWP 10765] [New LWP 10705] [New LWP 10706] [New LWP 10701] [New LWP 10712] [New LWP 10735] [New LWP 10713] [New LWP 10750] [New LWP 10718] [New LWP 10711] [New LWP 10716] [New LWP 10715] [New LWP 10785] [New LWP 10766] [New LWP 10796] [New LWP 10720] [New LWP 10725] [New LWP 10736] [New LWP 10709] [New LWP 10730] [New LWP 11541] [New LWP 10770] [New LWP 11573] [New LWP 10778] [New LWP 10804] [New LWP 11561] [New LWP 9388] [New LWP 9398] [New LWP 11538] [New LWP 10790] [New LWP 11586] [New LWP 10798] [New LWP 9910] [New LWP 10726] [New LWP 21823] [New LWP 10815] [New LWP 9397] [New LWP 11248] [New LWP 10723] [New LWP 11253] [New LWP 10728] [New LWP 10791] [New LWP 9389] [New LWP 10724] [New LWP 10780] [New LWP 11287] [New LWP 11592] [New LWP 10816] [New LWP 10812] [New LWP 10787] [New LWP 20622] [New LWP 21822] [New LWP 10751] [New LWP 10768] [New LWP 10767] [New LWP 11874] [New LWP 10733] [New LWP 10811] [New LWP 11574] [New LWP 11873] [New LWP 10771] [New LWP 11551] [New LWP 10799] [New LWP 10729] [New LWP 18254] [New LWP 10792] [New LWP 10803] [New LWP 9912] [New LWP 11293] [New LWP 20623] [New LWP 14805] [New LWP 10773] [New LWP 11298] [New LWP 11872] [New LWP 10763] [New LWP 10783] [New LWP 10769] [New LWP 11300] [New LWP 10777] [New LWP 10764] [New LWP 10802] [New LWP 10749] [New LWP 14806] [New LWP 10806] [New LWP 10805] [New LWP 18255] [New LWP 10181] [New LWP 11277] [New LWP 9913] [New LWP 10800] [New LWP 10801] [New LWP 11555] [New LWP 11871] [New LWP 10748] [New LWP 9915] [New LWP 10779] [New LWP 11294] [New LWP 9916] [New LWP 10757] [New LWP 10734] [New LWP 10786] [New LWP 10727] [New LWP 19063] [New LWP 11279] [New LWP 9905] [New LWP 9911] [New LWP 10772] [New LWP 10722] [New LWP 9914] [New LWP 10789] [New LWP 11540] [New LWP 9917] [New LWP 11289] [New LWP 10714] [New LWP 10721] [New LWP 10719] [New LWP 10788] [New LWP 10782] [New LWP 10784] [New LWP 10776] [New LWP 10774] [New LWP 10737] [New LWP 19064] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/bin/ceph-osd -i 1 --pid-file /var/run/ceph/osd.1.pid -c /etc/ceph/ceph.con'. Program terminated with signal 6, Aborted. #0 0x00007ff9ad91eb7b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0 (gdb) Thread 135 (Thread 0x7ff99a492700 (LWP 19064)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x108cd110) at ./common/Cond.h:55 #2 Pipe::writer (this=0x108ccf00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 134 (Thread 0x7ff975e1d700 (LWP 10737)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a53180) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a53180, buf=<optimized out>, buf@entry=0x7ff975e1cccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a53180) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 133 (Thread 0x7ff972dda700 (LWP 10774)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc42780) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc42780, buf=<optimized out>, buf@entry=0x7ff972dd9ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc42780) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 132 (Thread 0x7ff972cd9700 (LWP 10776)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc43390) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc43180) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 131 (Thread 0x7ff9725d2700 (LWP 10784)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc43d90) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc43b80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 130 (Thread 0x7ff9727d4700 (LWP 10782)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc42210) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc42000) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 129 (Thread 0x7ff9721ce700 (LWP 10788)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbe210) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbe000) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 128 (Thread 0x7ff97c438700 (LWP 10719)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be34e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be34e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3470, wt=0x3140bd0) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 127 (Thread 0x7ff97b436700 (LWP 10721)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be3808) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be3808) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3790, wt=0x3140900) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 126 (Thread 0x7ff97ec3d700 (LWP 10714)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be34e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be34e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3470, wt=0x315e5d0) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 125 (Thread 0x7ff9528db700 (LWP 11289)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc42e90) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc42c80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 124 (Thread 0x7ff9a49b1700 (LWP 9917)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b7be92 in Wait (mutex=..., this=0x4be5628) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x4be5618) at common/Timer.cc:112 #3 0x0000000000b7d44d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 123 (Thread 0x7ff9612b1700 (LWP 11540)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2c302990) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2c302780) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 122 (Thread 0x7ff9720cd700 (LWP 10789)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbf110) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbef00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 121 (Thread 0x7ff9a88fd700 (LWP 9914)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b6e1e8 in Wait (mutex=..., this=0x1ca5148) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1ca4c00) at msg/SimpleMessenger.cc:210 #3 0x0000000000b735ad in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 120 (Thread 0x7ff97ac35700 (LWP 10722)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be3998) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be3998) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3920, wt=0x3140330) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 119 (Thread 0x7ff972fdc700 (LWP 10772)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc42000) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc42000, buf=<optimized out>, buf@entry=0x7ff972fdbccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc42000) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 118 (Thread 0x7ff9a61b4700 (LWP 9911)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b6e1e8 in Wait (mutex=..., this=0x1ca4b48) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1ca4600) at msg/SimpleMessenger.cc:210 #3 0x0000000000b735ad in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 117 (Thread 0x7ff9a90fe700 (LWP 9905)): #0 0x00007ff9ac4f925e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ff9addda87d in ?? () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #2 0x00007ff9addd5b26 in leveldb::TableBuilder::WriteRawBlock(leveldb::Slice const&, leveldb::CompressionType, leveldb::BlockHandle*) () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #3 0x00007ff9addd5d29 in leveldb::TableBuilder::WriteBlock(leveldb::BlockBuilder*, leveldb::BlockHandle*) () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #4 0x00007ff9addd5ec4 in leveldb::TableBuilder::Flush() () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #5 0x00007ff9addd606b in leveldb::TableBuilder::Add(leveldb::Slice const&, leveldb::Slice const&) () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #6 0x00007ff9addb4c59 in leveldb::BuildTable(std::string const&, leveldb::Env*, leveldb::Options const&, leveldb::TableCache*, leveldb::Iterator*, leveldb::FileMetaData*) () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #7 0x00007ff9addba8ff in leveldb::DBImpl::WriteLevel0Table(leveldb::MemTable*, leveldb::VersionEdit*, leveldb::Version*) () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #8 0x00007ff9addbae54 in leveldb::DBImpl::CompactMemTable() () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #9 0x00007ff9addbcdbd in leveldb::DBImpl::BackgroundCompaction() () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #10 0x00007ff9addbd87d in leveldb::DBImpl::BackgroundCall() () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #11 0x00007ff9addda3bf in ?? () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #12 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #13 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #14 0x0000000000000000 in ?? () Thread 116 (Thread 0x7ff952bde700 (LWP 11279)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2c303680) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2c303680, buf=<optimized out>, buf@entry=0x7ff952bddccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2c303680) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 115 (Thread 0x7ff9a92b3700 (LWP 19063)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x108ccf00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x108ccf00, buf=<optimized out>, buf@entry=0x7ff9a92b2ccf "\377\070}Ù\371\177", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x108ccf00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 114 (Thread 0x7ff978430700 (LWP 10727)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a52c10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a52a00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 113 (Thread 0x7ff9723d0700 (LWP 10786)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbe710) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbe500) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 112 (Thread 0x7ff976a2b700 (LWP 10734)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a52e90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a52c80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 111 (Thread 0x7ff9740e6700 (LWP 10757)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000ab5217 in SignalHandler::entry (this=0x616dc80) at global/signal_handler.cc:213 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 110 (Thread 0x7ff9a51b2700 (LWP 9916)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b7c518 in WaitUntil (when=..., mutex=..., this=0x4be3080) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x4be3070) at common/Timer.cc:114 #3 0x0000000000b7d44d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 109 (Thread 0x7ff951d9a700 (LWP 11294)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2c303d90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2c303b80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 108 (Thread 0x7ff972ad7700 (LWP 10779)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc42990) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc42780) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 107 (Thread 0x7ff9a80fc700 (LWP 9915)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b6e1e8 in Wait (mutex=..., this=0x1ca5748) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1ca5200) at msg/SimpleMessenger.cc:210 #3 0x0000000000b735ad in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 106 (Thread 0x7ff9748f0700 (LWP 10748)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a53610) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a53400) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 105 (Thread 0x7ff953a28700 (LWP 11871)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2e4cf400) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2e4cf400, buf=<optimized out>, buf@entry=0x7ff953a27ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2e4cf400) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 104 (Thread 0x7ff96080e700 (LWP 11555)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2c302500) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2c302500, buf=<optimized out>, buf@entry=0x7ff96080dccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2c302500) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 103 (Thread 0x7ff9718c5700 (LWP 10801)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc43900) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc43900, buf=<optimized out>, buf@entry=0x7ff9718c4ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc43900) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 102 (Thread 0x7ff9719c6700 (LWP 10800)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbe000) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbe000, buf=<optimized out>, buf@entry=0x7ff9719c5ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbe000) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 101 (Thread 0x7ff9a69b5700 (LWP 9913)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b6e1e8 in Wait (mutex=..., this=0x1ca6348) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1ca5e00) at msg/SimpleMessenger.cc:210 #3 0x0000000000b735ad in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 100 (Thread 0x7ff952add700 (LWP 11277)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc42c80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc42c80, buf=<optimized out>, buf@entry=0x7ff952adcccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc42c80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 99 (Thread 0x7ff9a2a86700 (LWP 10181)): #0 0x00007ff9ad91d89c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ff9ad919065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007ff9ad918eba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x0000000000b48ceb in Mutex::Lock (this=0xdaf7250, no_lockdep=<optimized out>) at common/Mutex.cc:89 #4 0x00000000009ff908 in KeyValueStore::_do_op (this=0x1c90000, osr=0xdaf7180, handle=...) at os/KeyValueStore.cc:1041 #5 0x0000000000b80a3a in ThreadPool::worker (this=0x1c90498, wt=0x5236690) at common/WorkQueue.cc:128 #6 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #7 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #8 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x0000000000000000 in ?? () Thread 98 (Thread 0x7ff952ef5700 (LWP 18255)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x32e74990) at ./common/Cond.h:55 #2 Pipe::writer (this=0x32e74780) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 97 (Thread 0x7ff9714c1700 (LWP 10805)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbe280) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbe280, buf=<optimized out>, buf@entry=0x7ff9714c0ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbe280) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 96 (Thread 0x7ff9713c0700 (LWP 10806)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbe500) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbe500, buf=<optimized out>, buf@entry=0x7ff9713bfccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbe500) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 95 (Thread 0x7ff9558fe700 (LWP 14806)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x18a85b10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x18a85900) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 94 (Thread 0x7ff9747ef700 (LWP 10749)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a53890) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a53680) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 93 (Thread 0x7ff9717c4700 (LWP 10802)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbec80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbec80, buf=<optimized out>, buf@entry=0x7ff9717c3ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbec80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 92 (Thread 0x7ff9738e5700 (LWP 10764)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a52f00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a52f00, buf=<optimized out>, buf@entry=0x7ff9738e4ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a52f00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 91 (Thread 0x7ff972bd8700 (LWP 10777)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc43400) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc43400, buf=<optimized out>, buf@entry=0x7ff972bd7ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc43400) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 90 (Thread 0x7ff951b98700 (LWP 11300)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2c303b80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2c303b80, buf=<optimized out>, buf@entry=0x7ff951b97ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2c303b80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 89 (Thread 0x7ff9732df700 (LWP 10769)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a53d90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a53b80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 88 (Thread 0x7ff9726d3700 (LWP 10783)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc43890) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc43680) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 87 (Thread 0x7ff9737e4700 (LWP 10763)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a53900) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a53900, buf=<optimized out>, buf@entry=0x7ff9737e3ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a53900) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 86 (Thread 0x7ff953927700 (LWP 11872)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2e4cf900) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2e4cf900, buf=<optimized out>, buf@entry=0x7ff953926ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2e4cf900) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 85 (Thread 0x7ff951c99700 (LWP 11298)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2c302000) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2c302000, buf=<optimized out>, buf@entry=0x7ff951c98ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2c302000) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 84 (Thread 0x7ff972edb700 (LWP 10773)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc43180) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc43180, buf=<optimized out>, buf@entry=0x7ff972edaccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc43180) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 83 (Thread 0x7ff99c214700 (LWP 14805)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x18a85900) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x18a85900, buf=<optimized out>, buf@entry=0x7ff99c213ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x18a85900) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 82 (Thread 0x7ff99a391700 (LWP 20623)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x108cdd90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x108cdb80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 81 (Thread 0x7ff951e9b700 (LWP 11293)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2c302210) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2c302000) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 80 (Thread 0x7ff9a71b6700 (LWP 9912)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b6e1e8 in Wait (mutex=..., this=0x1ca5d48) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1ca5800) at msg/SimpleMessenger.cc:210 #3 0x0000000000b735ad in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 79 (Thread 0x7ff9716c3700 (LWP 10803)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc43680) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc43680, buf=<optimized out>, buf@entry=0x7ff9716c2ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc43680) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 78 (Thread 0x7ff971dca700 (LWP 10792)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbe990) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbe780) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 77 (Thread 0x7ff952ff6700 (LWP 18254)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x32e74780) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x32e74780, buf=<optimized out>, buf@entry=0x7ff952ff5ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x32e74780) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 76 (Thread 0x7ff977b2e700 (LWP 10729)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b7be92 in Wait (mutex=..., this=0x4be4f60) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x4be4f50) at common/Timer.cc:112 #3 0x0000000000b7d44d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 75 (Thread 0x7ff971ac7700 (LWP 10799)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbe780) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbe780, buf=<optimized out>, buf@entry=0x7ff971ac6ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbe780) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 74 (Thread 0x7ff9a72ee700 (LWP 11551)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2c302780) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2c302780, buf=<optimized out>, buf@entry=0x7ff9a72edccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2c302780) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 73 (Thread 0x7ff9730dd700 (LWP 10771)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc42a00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc42a00, buf=<optimized out>, buf@entry=0x7ff9730dcccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc42a00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 72 (Thread 0x7ff9535ff700 (LWP 11873)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2e4cfb10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e4cf900) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 71 (Thread 0x7ff95ba9e700 (LWP 11574)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2e4cef00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2e4cef00, buf=<optimized out>, buf@entry=0x7ff95ba9dccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2e4cef00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 70 (Thread 0x7ff96a378700 (LWP 10811)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc42500) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc42500, buf=<optimized out>, buf@entry=0x7ff96a377ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc42500) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 69 (Thread 0x7ff976b2c700 (LWP 10733)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a52a00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a52a00, buf=<optimized out>, buf@entry=0x7ff976b2bccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a52a00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 68 (Thread 0x7ff9534fe700 (LWP 11874)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2e4cf610) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e4cf400) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 67 (Thread 0x7ff9734e1700 (LWP 10767)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a53b80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a53b80, buf=<optimized out>, buf@entry=0x7ff9734e0ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a53b80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 66 (Thread 0x7ff9733e0700 (LWP 10768)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc42280) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc42280, buf=<optimized out>, buf@entry=0x7ff9733dfccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc42280) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 65 (Thread 0x7ff9745ed700 (LWP 10751)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a53680) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a53680, buf=<optimized out>, buf@entry=0x7ff9745ecccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a53680) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 64 (Thread 0x7ff996594700 (LWP 21822)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x19718c80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x19718c80, buf=<optimized out>, buf@entry=0x7ff996593ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x19718c80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 63 (Thread 0x7ff9a3eab700 (LWP 20622)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x108cdb80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x108cdb80, buf=<optimized out>, buf@entry=0x7ff9a3eaaccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x108cdb80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 62 (Thread 0x7ff9722cf700 (LWP 10787)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbe490) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbe280) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 61 (Thread 0x7ff96a277700 (LWP 10812)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc42f00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc42f00, buf=<optimized out>, buf@entry=0x7ff96a276ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc42f00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 60 (Thread 0x7ff96a075700 (LWP 10816)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc43110) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc42f00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 59 (Thread 0x7ff9562bd700 (LWP 11592)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2c303610) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2c303400) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 58 (Thread 0x7ff9529dc700 (LWP 11287)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2c303890) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2c303680) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 57 (Thread 0x7ff9728d5700 (LWP 10780)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc43610) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc43400) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 56 (Thread 0x7ff979c33700 (LWP 10724)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ad4ce2 in Wait (mutex=..., this=0x4be57a0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x4be5740) at common/Finisher.cc:78 #3 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 55 (Thread 0x7ff9ab755700 (LWP 9389)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ae3ecb in ceph::log::Log::entry (this=0x1c86000) at log/Log.cc:323 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 54 (Thread 0x7ff971ecb700 (LWP 10791)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbec10) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbea00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 53 (Thread 0x7ff97832f700 (LWP 10728)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b7be92 in Wait (mutex=..., this=0x4be5500) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x4be54f0) at common/Timer.cc:112 #3 0x0000000000b7d44d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 52 (Thread 0x7ff952cdf700 (LWP 11253)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2c302c80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2c302c80, buf=<optimized out>, buf@entry=0x7ff952cdeccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2c302c80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 51 (Thread 0x7ff97a434700 (LWP 10723)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x000000000075d422 in WaitUntil (when=..., mutex=..., this=0x4be3b40) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be3b40) at ./common/Cond.h:79 #3 OSD::heartbeat_entry (this=0x4be3000) at osd/OSD.cc:3094 #4 0x00000000007bd65d in OSD::T_Heartbeat::entry (this=<optimized out>) at osd/OSD.h:1080 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 50 (Thread 0x7ff952de0700 (LWP 11248)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2c302e90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2c302c80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 49 (Thread 0x7ff9aa7ef700 (LWP 9397)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ba21c2 in WaitUntil (when=..., mutex=..., this=0x1c65188) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1c65188) at ./common/Cond.h:79 #3 CephContextServiceThread::entry (this=0x1c65110) at common/ceph_context.cc:58 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 48 (Thread 0x7ff96a176700 (LWP 10815)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc42710) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc42500) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 47 (Thread 0x7ff995bc0700 (LWP 21823)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x19718e90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x19718c80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 46 (Thread 0x7ff978c31700 (LWP 10726)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b7c518 in WaitUntil (when=..., mutex=..., this=0x4be5080) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x4be5070) at common/Timer.cc:114 #3 0x0000000000b7d44d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 45 (Thread 0x7ff9a59b3700 (LWP 9910)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b6e1e8 in Wait (mutex=..., this=0x1ca4548) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1ca4000) at msg/SimpleMessenger.cc:210 #3 0x0000000000b735ad in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 44 (Thread 0x7ff971bc8700 (LWP 10798)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbef00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbef00, buf=<optimized out>, buf@entry=0x7ff971bc7ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbef00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 43 (Thread 0x7ff957a6c700 (LWP 11586)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2e4cf110) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e4cef00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 42 (Thread 0x7ff971fcc700 (LWP 10790)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbee90) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbec80) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 41 (Thread 0x7ff9a3597700 (LWP 11538)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdcbfb10) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdcbf900) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 40 (Thread 0x7ff9a9fee700 (LWP 9398)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000b8612d in AdminSocket::entry (this=0x1c86140) at common/admin_socket.cc:239 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 39 (Thread 0x7ff9ae3dd780 (LWP 9388)): #0 0x00007ff9ad918148 in pthread_join () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b76f52 in Thread::join (this=0x1ca4280, prval=<optimized out>) at common/Thread.cc:140 #2 0x0000000000b6dd03 in SimpleMessenger::wait (this=0x1ca4000) at msg/SimpleMessenger.cc:530 #3 0x000000000073199b in main (argc=<optimized out>, argv=<optimized out>) at ceph_osd.cc:522 Thread 38 (Thread 0x7ff95ca12700 (LWP 11561)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbf900) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbf900, buf=<optimized out>, buf@entry=0x7ff95ca11ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbf900) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 37 (Thread 0x7ff9715c2700 (LWP 10804)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdc43b80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdc43b80, buf=<optimized out>, buf@entry=0x7ff9715c1ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdc43b80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 36 (Thread 0x7ff9729d6700 (LWP 10778)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc42c10) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc42a00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 35 (Thread 0x7ff95c464700 (LWP 11573)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x2c303400) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x2c303400, buf=<optimized out>, buf@entry=0x7ff95c463ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x2c303400) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 34 (Thread 0x7ff9731de700 (LWP 10770)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc42490) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc42280) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 33 (Thread 0x7ff9611b0700 (LWP 11541)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x2c302710) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2c302500) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 32 (Thread 0x7ff97732d700 (LWP 10730)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00000000007411c5 in Wait (mutex=..., this=<optimized out>) at ./common/Cond.h:55 #2 OSDService::agent_entry (this=0x4be46e8) at osd/OSD.cc:500 #3 0x00000000007a03fd in OSDService::AgentThread::entry (this=<optimized out>) at osd/OSD.h:501 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 31 (Thread 0x7ff981442700 (LWP 10709)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c1e579 in Accepter::entry (this=0x1ca52c0) at msg/Accepter.cc:196 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 30 (Thread 0x7ff976122700 (LWP 10736)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a53390) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a53180) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 29 (Thread 0x7ff979432700 (LWP 10725)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ad4ce2 in Wait (mutex=..., this=0x4be53a0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x4be5340) at common/Finisher.cc:78 #3 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 28 (Thread 0x7ff97bc37700 (LWP 10720)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be3678) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be3678) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3600, wt=0x31419e0) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 27 (Thread 0x7ff971cc9700 (LWP 10796)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0xdcbea00) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0xdcbea00, buf=<optimized out>, buf@entry=0x7ff971cc8ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0xdcbea00) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 26 (Thread 0x7ff9735e2700 (LWP 10766)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a53b10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a53900) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 25 (Thread 0x7ff9724d1700 (LWP 10785)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0xdc43b10) at ./common/Cond.h:55 #2 Pipe::writer (this=0xdc43900) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 24 (Thread 0x7ff97e43c700 (LWP 10715)): #0 0x00007ff9ad91d89c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007ff9ad919065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007ff9ad918eba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x0000000000b48ceb in Mutex::Lock (this=0x1cde348, no_lockdep=<optimized out>) at common/Mutex.cc:89 #4 0x0000000000a003b3 in Locker (m=..., this=0x7ff97e43a360) at ./common/Mutex.h:120 #5 GenericObjectMap::lookup_header (this=this@entry=0x1cde340, cid=..., oid=...) at os/GenericObjectMap.h:211 #6 0x00000000009ec659 in StripObjectMap::lookup_strip_header (this=0x1cde340, cid=..., oid=..., strip_header=strip_header@entry=0x7ff97e43a5b0) at os/KeyValueStore.cc:115 #7 0x00000000009ef252 in KeyValueStore::getattr (this=0x1c90000, cid=..., oid=..., name=name@entry=0x2987d3d8 "_", bp=...) at os/KeyValueStore.cc:1975 #8 0x0000000000988350 in PGBackend::objects_get_attr (this=0x9cd1cc0, hoid=..., attr=..., out=0x7ff97e43a8f0) at osd/PGBackend.cc:188 #9 0x00000000009188f0 in ReplicatedPG::get_object_context (this=this@entry=0xb02ac00, soid=..., can_create=can_create@entry=false, attrs=attrs@entry=0x0) at osd/ReplicatedPG.cc:7122 #10 0x0000000000925c00 in ReplicatedPG::find_object_context (this=this@entry=0xb02ac00, oid=..., pobc=pobc@entry=0x7ff97e43afc0, can_create=can_create@entry=false, map_snapid_to_clone=false, pmissing=pmissing@entry=0x7ff97e43b310) at osd/ReplicatedPG.cc:7253 #11 0x000000000094cb8b in ReplicatedPG::do_op (this=this@entry=0xb02ac00, op=...) at osd/ReplicatedPG.cc:1322 #12 0x00000000008e522b in ReplicatedPG::do_request (this=this@entry=0xb02ac00, op=..., handle=...) at osd/ReplicatedPG.cc:1129 #13 0x000000000073fdf6 in OSD::dequeue_op (this=0x4be3000, pg=..., op=..., handle=...) at osd/OSD.cc:7780 #14 0x000000000075dc5a in OSD::OpWQ::_process (this=0x4be3e58, pg=..., handle=...) at osd/OSD.cc:7750 #15 0x000000000079a42e in ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process (this=0x4be3e58, handle=...) at ./common/WorkQueue.h:191 #16 0x0000000000b80a3a in ThreadPool::worker (this=0x4be3470, wt=0x315e570) at common/WorkQueue.cc:128 #17 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #18 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #19 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #20 0x0000000000000000 in ?? () Thread 23 (Thread 0x7ff97dc3b700 (LWP 10716)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be34e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be34e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3470, wt=0x315f9b0) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 22 (Thread 0x7ff980440700 (LWP 10711)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ad4ce2 in Wait (mutex=..., this=0x7ffff31bdf00) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x7ffff31bdea0) at common/Finisher.cc:78 #3 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 21 (Thread 0x7ff97cc39700 (LWP 10718)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be34e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be34e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3470, wt=0x31409c0) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 20 (Thread 0x7ff9746ee700 (LWP 10750)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a53400) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a53400, buf=<optimized out>, buf@entry=0x7ff9746edccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a53400) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 19 (Thread 0x7ff97f43e700 (LWP 10713)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be34e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be34e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3470, wt=0x3140cc0) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 18 (Thread 0x7ff97692a700 (LWP 10735)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c3e73c in Pipe::tcp_read_wait (this=this@entry=0x4a52c80) at msg/Pipe.cc:2282 #2 0x0000000000c3e9d0 in Pipe::tcp_read (this=this@entry=0x4a52c80, buf=<optimized out>, buf@entry=0x7ff976929ccf "\377", len=len@entry=1) at msg/Pipe.cc:2255 #3 0x0000000000c5095f in Pipe::reader (this=0x4a52c80) at msg/Pipe.cc:1421 #4 0x0000000000c5497d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 17 (Thread 0x7ff97fc3f700 (LWP 10712)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be34e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be34e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3470, wt=0x3141c50) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 16 (Thread 0x7ff98544a700 (LWP 10701)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c2ffb3 in Wait (mutex=..., this=0x1ca4d60) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1ca4cf8) at msg/DispatchQueue.cc:138 #3 0x0000000000b729bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 15 (Thread 0x7ff982c45700 (LWP 10706)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c2ffb3 in Wait (mutex=..., this=0x1ca5f60) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1ca5ef8) at msg/DispatchQueue.cc:138 #3 0x0000000000b729bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 14 (Thread 0x7ff983446700 (LWP 10705)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c1e579 in Accepter::entry (this=0x1ca58c0) at msg/Accepter.cc:196 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 13 (Thread 0x7ff9736e3700 (LWP 10765)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c496da in Wait (mutex=..., this=0x4a53110) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4a52f00) at msg/Pipe.cc:1730 #3 0x0000000000c5485d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 12 (Thread 0x7ff97d43a700 (LWP 10717)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b80e1a in WaitUntil (when=..., mutex=..., this=0x4be34e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x4be34e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x4be3470, wt=0x3140300) at common/WorkQueue.cc:146 #4 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 11 (Thread 0x7ff985c4b700 (LWP 10700)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c1e579 in Accepter::entry (this=0x1ca40c0) at msg/Accepter.cc:196 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 10 (Thread 0x7ff980c41700 (LWP 10710)): #0 0x00007ff9ad91b0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000b7c518 in WaitUntil (when=..., mutex=..., this=0x7ffff31bdde8) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x7ffff31bddd8) at common/Timer.cc:114 #3 0x0000000000b7d44d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 9 (Thread 0x7ff982444700 (LWP 10707)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c1e579 in Accepter::entry (this=0x1ca5ec0) at msg/Accepter.cc:196 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 8 (Thread 0x7ff981c43700 (LWP 10708)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c2ffb3 in Wait (mutex=..., this=0x1ca5360) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1ca52f8) at msg/DispatchQueue.cc:138 #3 0x0000000000b729bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 7 (Thread 0x7ff984c49700 (LWP 10702)): #0 0x00007ff9ac498a13 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x0000000000c1e579 in Accepter::entry (this=0x1ca4cc0) at msg/Accepter.cc:196 #2 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x0000000000000000 in ?? () Thread 6 (Thread 0x7ff983c47700 (LWP 10704)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c2ffb3 in Wait (mutex=..., this=0x1ca5960) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1ca58f8) at msg/DispatchQueue.cc:138 #3 0x0000000000b729bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 5 (Thread 0x7ff984448700 (LWP 10703)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c2ffb3 in Wait (mutex=..., this=0x1ca4760) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1ca46f8) at msg/DispatchQueue.cc:138 #3 0x0000000000b729bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 4 (Thread 0x7ff9a1283700 (LWP 10184)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ad4ce2 in Wait (mutex=..., this=0x1c90118) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x1c900b8) at common/Finisher.cc:78 #3 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 3 (Thread 0x7ff98644c700 (LWP 10699)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000c2ffb3 in Wait (mutex=..., this=0x1ca4160) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1ca40f8) at msg/DispatchQueue.cc:138 #3 0x0000000000b729bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 2 (Thread 0x7ff9a1a84700 (LWP 10183)): #0 0x00007ff9ad91ad84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ad4ce2 in Wait (mutex=..., this=0x1c903b8) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x1c90358) at common/Finisher.cc:78 #3 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #4 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x0000000000000000 in ?? () Thread 1 (Thread 0x7ff9a2285700 (LWP 10182)): #0 0x00007ff9ad91eb7b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x0000000000ab4609 in reraise_fatal (signum=6) at global/signal_handler.cc:59 #2 handle_fatal_signal (signum=6) at global/signal_handler.cc:105 #3 <signal handler called> #4 0x00007ff9ac3e8405 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ff9ac3ebb5b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x00007ff9acce689d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x00007ff9acce4996 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x00007ff9acce49c3 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x00007ff9acce4bee in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x0000000000b8ec7a in ceph::__ceph_assert_fail (assertion=0xc71914 "0", file=<optimized out>, line=975, func=0xccc0e0 "GenericObjectMap::Header GenericObjectMap::lookup_parent(GenericObjectMap::Header)") at common/assert.cc:77 #11 0x0000000000a9fa42 in GenericObjectMap::lookup_parent (this=<optimized out>, input=...) at os/GenericObjectMap.cc:975 #12 0x0000000000a9fb8b in GenericObjectMap::GenericObjectMapIteratorImpl::init (this=this@entry=0x52d0930) at os/GenericObjectMap.cc:298 #13 0x0000000000aa0469 in GenericObjectMap::GenericObjectMapIteratorImpl::seek_to_first (this=0x52d0930) at os/GenericObjectMap.cc:327 #14 0x0000000000aa5122 in GenericObjectMap::rm_keys (this=0x1cde340, header=..., prefix=..., to_clear=..., t=...) at os/GenericObjectMap.cc:538 #15 0x00000000009e8d5e in KeyValueStore::BufferTransaction::remove_buffer_keys (this=this@entry=0x7ff9a22848e0, strip_header=..., prefix=..., keys=...) at os/KeyValueStore.cc:355 #16 0x00000000009f7d75 in KeyValueStore::_rmattr (this=this@entry=0x1c90000, cid=..., oid=..., name=name@entry=0x1378d9f8 "snapset", t=...) at os/KeyValueStore.cc:2083 #17 0x00000000009fe81d in KeyValueStore::_do_transaction (this=this@entry=0x1c90000, transaction=..., t=..., handle=handle@entry=0x7ff9a2284b80) at os/KeyValueStore.cc:1231 #18 0x00000000009ff879 in KeyValueStore::_do_transactions (this=this@entry=0x1c90000, tls=..., op_seq=<optimized out>, handle=handle@entry=0x7ff9a2284b80) at os/KeyValueStore.cc:1105 #19 0x00000000009ff977 in KeyValueStore::_do_op (this=0x1c90000, osr=<optimized out>, handle=...) at os/KeyValueStore.cc:1044 #20 0x0000000000b80a3a in ThreadPool::worker (this=0x1c90498, wt=0x5235b90) at common/WorkQueue.cc:128 #21 0x0000000000b81c80 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #22 0x00007ff9ad916e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #23 0x00007ff9ac4a43dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #24 0x0000000000000000 in ?? () (gdb) quit
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
