Hi Sage, We have 3 osd failed first, two of them have the same log as osd3 , the other one is osd 7. We just use it as a file system, I wrote a script to repeatedly write a 10G file and delete it. The only unusual thing I saw in the log was the journal file was fulled before the code trace log. The other thing is that when I try to restart the failed osd, more osd crashed on other node. However we didn't get the core dump :( Thanks. Regards, Leander Yu. On Tue, Sep 7, 2010 at 2:42 AM, Sage Weil <sage@xxxxxxxxxxxx> wrote: > Hi, > > This is one we've seen before, issue #326 > > http://tracker.newdream.net/issues/326 > > Was that the first (and only?) osd to fail? > > What kind of workload were you subjecting the cluster to? Just the file > system? RBD? Anything unusual? > > Also, can you confirm what version of the code you were running? The osd > log at /var/log/ceph/osd.*.log should have a version number and sha1 id, > something like > > ceph version 0.22~rc (3cd9d853cd58c79dc12427be8488e57970abda04) > > Thanks! > sage > > > On Mon, 6 Sep 2010, Leander Yu wrote: > >> Hi all, >> I have setup a 10 osd + 2 mds + 3 mon ceph cluster. it runs ok at >> beginning. However after one day, some of the osd crashed with >> following assert fail >> I am using the unstable trunk. ceph.conf is attached. >> >> -------------- osd 3 ----------------- >> osd/PG.h: In function 'void PG::IndexedLog::index(PG::Log::Entry&)': >> osd/PG.h:429: FAILED assert(caller_ops.count(e.reqid) == 0) >> 1: (OSD::_process_pg_info(unsigned int, int, PG::Info&, PG::Log&, >> PG::Missing&, std::map<int, MOSDPGInfo*, std::less<int>, >> std::allocator<std::pair<int const, MOSDPGInfo*> > >*, int&)+0xb06) >> [0x4cf426] >> 2: (OSD::handle_pg_log(MOSDPGLog*)+0xa9) [0x4cf999] >> 3: (OSD::_dispatch(Message*)+0x3ed) [0x4e7dfd] >> 4: (OSD::ms_dispatch(Message*)+0x39) [0x4e86c9] >> 5: (SimpleMessenger::dispatch_entry()+0x789) [0x46b5f9] >> 6: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x45849c] >> 7: (Thread::_entry_func(void*)+0xa) [0x46c0ca] >> 8: (()+0x6a3a) [0x7f69fd39ea3a] >> 9: (clone()+0x6d) [0x7f69fc5bc77d] >> >> -------------- osd 7 -------------------- >> osd/ReplicatedPG.cc: In function 'void ReplicatedPG::sub_op_pull(MOSDSubOp*)': >> osd/ReplicatedPG.cc:3021: FAILED assert(r == 0) >> 1: (OSD::dequeue_op(PG*)+0x344) [0x4e6fd4] >> 2: (ThreadPool::worker()+0x28f) [0x5b5a9f] >> 3: (ThreadPool::WorkThread::entry()+0xd) [0x4f0acd] >> 4: (Thread::_entry_func(void*)+0xa) [0x46c0ca] >> 5: (()+0x6a3a) [0x7efff4f12a3a] >> 6: (clone()+0x6d) [0x7efff413077d] >> >> Please let me if you need more information. I still keep the >> environment for collecting more data for debug. >> >> Thanks. >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
