That is very likely to be one of the merge_log bugs fixed between 0.48 and 0.55. I could confirm with a stacktrace from gdb with line numbers or the remainder of the logging dumped when the daemon crashed. My understanding of your situation is that currently all pgs are active+clean but you are missing some rbd image headers and some rbd images appear to be corrupted. Is that accurate? -Sam On Fri, Dec 7, 2012 at 6:39 AM, Oliver Francke <Oliver.Francke@xxxxxxxx> wrote: > Hi, > > is the following a "known one", too? Would be good to get it out of my head: > > >> /var/log/ceph/ceph-osd.40.log.1.gz: 1: /usr/bin/ceph-osd() [0x706c59] >> /var/log/ceph/ceph-osd.40.log.1.gz: 2: (()+0xeff0) [0x7f7f306c0ff0] >> /var/log/ceph/ceph-osd.40.log.1.gz: 3: (gsignal()+0x35) [0x7f7f2f35f1b5] >> /var/log/ceph/ceph-osd.40.log.1.gz: 4: (abort()+0x180) [0x7f7f2f361fc0] >> /var/log/ceph/ceph-osd.40.log.1.gz: 5: >> (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7f7f2fbf3dc5] >> /var/log/ceph/ceph-osd.40.log.1.gz: 6: (()+0xcb166) [0x7f7f2fbf2166] >> /var/log/ceph/ceph-osd.40.log.1.gz: 7: (()+0xcb193) [0x7f7f2fbf2193] >> /var/log/ceph/ceph-osd.40.log.1.gz: 8: (()+0xcb28e) [0x7f7f2fbf228e] >> /var/log/ceph/ceph-osd.40.log.1.gz: 9: (ceph::__ceph_assert_fail(char >> const*, char const*, int, char const*)+0x793) [0x77e903] >> /var/log/ceph/ceph-osd.40.log.1.gz: 10: >> (PG::merge_log(ObjectStore::Transaction&, pg_info_t&, pg_log_t&, >> int)+0x1de3) [0x63db93] >> /var/log/ceph/ceph-osd.40.log.1.gz: 11: >> (PG::RecoveryState::Stray::react(PG::RecoveryState::MLogRec const&)+0x2cc) >> [0x63e00c] >> /var/log/ceph/ceph-osd.40.log.1.gz: 12: >> (boost::statechart::simple_state<PG::RecoveryState::Stray, >> PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na>, >> (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base >> const&, void const*)+0x203) [0x658a63] >> /var/log/ceph/ceph-osd.40.log.1.gz: 13: >> (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, >> PG::RecoveryState::Initial, std::allocator<void>, >> boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base >> const&)+0x6b) [0x650b4b] >> /var/log/ceph/ceph-osd.40.log.1.gz: 14: >> (PG::RecoveryState::handle_log(int, MOSDPGLog*, PG::RecoveryCtx*)+0x190) >> [0x60a520] >> /var/log/ceph/ceph-osd.40.log.1.gz: 15: >> (OSD::handle_pg_log(std::tr1::shared_ptr<OpRequest>)+0x666) [0x5c62e6] >> /var/log/ceph/ceph-osd.40.log.1.gz: 16: >> (OSD::dispatch_op(std::tr1::shared_ptr<OpRequest>)+0x11b) [0x5c6f3b] >> /var/log/ceph/ceph-osd.40.log.1.gz: 17: (OSD::_dispatch(Message*)+0x173) >> [0x5d1983] >> /var/log/ceph/ceph-osd.40.log.1.gz: 18: (OSD::ms_dispatch(Message*)+0x184) >> [0x5d2254] >> /var/log/ceph/ceph-osd.40.log.1.gz: 19: >> (SimpleMessenger::DispatchQueue::entry()+0x5e9) [0x7d3c09] >> /var/log/ceph/ceph-osd.40.log.1.gz: 20: >> (SimpleMessenger::dispatch_entry()+0x15) [0x7d5195] >> /var/log/ceph/ceph-osd.40.log.1.gz: 21: >> (SimpleMessenger::DispatchThread::entry()+0xd) [0x726bad] >> /var/log/ceph/ceph-osd.40.log.1.gz: 22: (()+0x68ca) [0x7f7f306b88ca] >> /var/log/ceph/ceph-osd.40.log.1.gz: 23: (clone()+0x6d) [0x7f7f2f3fc92d] >> > > Thnx for looking, > > > Oliver. > > -- > > Oliver Francke > > filoo GmbH > Moltkestraße 25a > 33330 Gütersloh > HRB4355 AG Gütersloh > > Geschäftsführer: S.Grewing | J.Rehpöhler | C.Kunz > > Folgen Sie uns auf Twitter: http://twitter.com/filoogmbh > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
