Hi, This is one we've seen before, issue #326 http://tracker.newdream.net/issues/326 Was that the first (and only?) osd to fail? What kind of workload were you subjecting the cluster to? Just the file system? RBD? Anything unusual? Also, can you confirm what version of the code you were running? The osd log at /var/log/ceph/osd.*.log should have a version number and sha1 id, something like ceph version 0.22~rc (3cd9d853cd58c79dc12427be8488e57970abda04) Thanks! sage On Mon, 6 Sep 2010, Leander Yu wrote: > Hi all, > I have setup a 10 osd + 2 mds + 3 mon ceph cluster. it runs ok at > beginning. However after one day, some of the osd crashed with > following assert fail > I am using the unstable trunk. ceph.conf is attached. > > -------------- osd 3 ----------------- > osd/PG.h: In function 'void PG::IndexedLog::index(PG::Log::Entry&)': > osd/PG.h:429: FAILED assert(caller_ops.count(e.reqid) == 0) > 1: (OSD::_process_pg_info(unsigned int, int, PG::Info&, PG::Log&, > PG::Missing&, std::map<int, MOSDPGInfo*, std::less<int>, > std::allocator<std::pair<int const, MOSDPGInfo*> > >*, int&)+0xb06) > [0x4cf426] > 2: (OSD::handle_pg_log(MOSDPGLog*)+0xa9) [0x4cf999] > 3: (OSD::_dispatch(Message*)+0x3ed) [0x4e7dfd] > 4: (OSD::ms_dispatch(Message*)+0x39) [0x4e86c9] > 5: (SimpleMessenger::dispatch_entry()+0x789) [0x46b5f9] > 6: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x45849c] > 7: (Thread::_entry_func(void*)+0xa) [0x46c0ca] > 8: (()+0x6a3a) [0x7f69fd39ea3a] > 9: (clone()+0x6d) [0x7f69fc5bc77d] > > -------------- osd 7 -------------------- > osd/ReplicatedPG.cc: In function 'void ReplicatedPG::sub_op_pull(MOSDSubOp*)': > osd/ReplicatedPG.cc:3021: FAILED assert(r == 0) > 1: (OSD::dequeue_op(PG*)+0x344) [0x4e6fd4] > 2: (ThreadPool::worker()+0x28f) [0x5b5a9f] > 3: (ThreadPool::WorkThread::entry()+0xd) [0x4f0acd] > 4: (Thread::_entry_func(void*)+0xa) [0x46c0ca] > 5: (()+0x6a3a) [0x7efff4f12a3a] > 6: (clone()+0x6d) [0x7efff413077d] > > Please let me if you need more information. I still keep the > environment for collecting more data for debug. > > Thanks. >
