Hi,
I have a small cluster with only three nodes, 4 OSDs + 3 OSDs. I have been running version 0.87.2 (Giant) for over 2.5 year, but a couple of day ago I upgraded to 0.94.10 (Hammer) and then up to 10.2.7 (Jewel). Both the upgrades went great. Started with monitors, osd and finally mds. The log shows all 448 pgs active+clean. I'm running all daemons inside docker and ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
Today I had a power outage and I had to take down the servers. When I now start the servers again one OSD daemon doesn't start properly. It keeps crashing.
I noticed that the two first restarts of the osd daemon crashed with this error:
FAILED assert(rollback_info_trimmed_to_riter == log.rbegin())
After that it always fails with "FAILED assert(i.first <= i.last)"
I have 15 logs like this one:
Jun 18 08:56:18 island sh[27991]: 2017-06-18 08:56:18.300641 7f5c5e0ff8c0 -1 log_channel(cluster) log [ERR] : 2.38 log bound mismatch, info (19544'666742,19691'671046] actual [19499'665843,19691'671046]
I removed the directories <pg_id>_head, but that just removed these error logs. It crashes anyway.
Anyone has any suggestions what to do to make it start up correct. Of course I can remove the OSD from the cluster and re-add it, but it feels like a bug.
A small snippet from the logs is added below. I didn't include the event list. If it will help I can send it too.
Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: In function 'static bool pg_interval_t::check_new_interval(int, int, const std::vector<int>&, const std::vector<int>&, int, int, const std::vector<int>&, const std::vector<int>&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t, IsPGRecoverablePredicate*, std::map<unsigned int, pg_interval_t>*, std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED assert(i.first <= i.last)
Jun 18 13:52:23 island sh[7068]: ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
Jun 18 13:52:23 island sh[7068]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x559fe4c14360]
Jun 18 13:52:23 island sh[7068]: 2: (pg_interval_t::check_new_interval(int, int, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, int, int, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, unsigned int, unsigned int, std::shared_ptr<OSDMap const>, std::shared_ptr<OSDMap const>, pg_t, IsPGRecoverablePredicate*, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > >*, std::ostream*)+0x72c) [0x559fe47f723c]
Jun 18 13:52:23 island sh[7068]: 3: (PG::start_peering_interval(std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> > const&, int, std::vector<int, std::allocator<int> > const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
Jun 18 13:52:23 island sh[7068]: 4: (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478) [0x559fe4615828]
Jun 18 13:52:23 island sh[7068]: 5: (boost::statechart::simple_state<PG::RecoveryState::Reset, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x176) [0x559fe4645b86]
Jun 18 13:52:23 island sh[7068]: 6: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x69) [0x559fe4626d49]
Jun 18 13:52:23 island sh[7068]: 7: (PG::handle_advance_map(std::shared_ptr<OSDMap const>, std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, int, std::vector<int, std::allocator<int> >&, int, PG::RecoveryCtx*)+0x49e) [0x559fe45fa5ae]
Jun 18 13:52:23 island sh[7068]: 8: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x2f2) [0x559fe452c042]
Jun 18 13:52:23 island sh[7068]: 9: (OSD::process_peering_events(std::__cxx11::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x214) [0x559fe4546d34]
Jun 18 13:52:23 island sh[7068]: 10: (ThreadPool::BatchWorkQueue<PG>::_void_process(void*, ThreadPool::TPHandle&)+0x25) [0x559fe458f8e5]
Jun 18 13:52:23 island sh[7068]: 11: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x559fe4c06531]
Jun 18 13:52:23 island sh[7068]: 12: (ThreadPool::WorkThread::entry()+0x10) [0x559fe4c07630]
Jun 18 13:52:23 island sh[7068]: 13: (()+0x76fa) [0x7f4fe256b6fa]
Jun 18 13:52:23 island sh[7068]: 14: (clone()+0x6d) [0x7f4fe05e3b5d]
Jun 18 13:52:23 island sh[7068]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jun 18 13:52:23 island sh[7068]: 2017-06-18 13:52:23.599558 7f4fc2500700 -1 osd/osd_types.cc: In function 'static bool pg_interval_t::check_new_interval(int, int, const std::vector<int>&, const std::vector<int>&, int, int, const std::vector<int>&, const std::vector<int>&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t, IsPGRecoverablePredicate*, std::map<unsigned int, pg_interval_t>*, std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED assert(i.first <= i.last)
Jun 18 13:52:23 island sh[7068]: ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
Jun 18 13:52:23 island sh[7068]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x559fe4c14360]
Jun 18 13:52:23 island sh[7068]: 2: (pg_interval_t::check_new_interval(int, int, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, int, int, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, unsigned int, unsigned int, std::shared_ptr<OSDMap const>, std::shared_ptr<OSDMap const>, pg_t, IsPGRecoverablePredicate*, std::map<unsigned int, pg_interval_t, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, pg_interval_t> > >*, std::ostream*)+0x72c) [0x559fe47f723c]
Jun 18 13:52:23 island sh[7068]: 3: (PG::start_peering_interval(std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> > const&, int, std::vector<int, std::allocator<int> > const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
Jun 18 13:52:23 island sh[7068]: 4: (PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478) [0x559fe4615828]
Jun 18 13:52:23 island sh[7068]: 5: (boost::statechart::simple_state<PG::RecoveryState::Reset, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x176) [0x559fe4645b86]
Jun 18 13:52:23 island sh[7068]: 6: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x69) [0x559fe4626d49]
Jun 18 13:52:23 island sh[7068]: 7: (PG::handle_advance_map(std::shared_ptr<OSDMap const>, std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, int, std::vector<int, std::allocator<int> >&, int, PG::RecoveryCtx*)+0x49e) [0x559fe45fa5ae]
Jun 18 13:52:23 island sh[7068]: 8: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set<boost::intrusive_ptr<PG>, std::less<boost::intrusive_ptr<PG> >, std::allocator<boost::intrusive_ptr<PG> > >*)+0x2f2) [0x559fe452c042]
Jun 18 13:52:23 island sh[7068]: 9: (OSD::process_peering_events(std::__cxx11::list<PG*, std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x214) [0x559fe4546d34]
Jun 18 13:52:23 island sh[7068]: 10: (ThreadPool::BatchWorkQueue<PG>::_void_process(void*, ThreadPool::TPHandle&)+0x25) [0x559fe458f8e5]
Jun 18 13:52:23 island sh[7068]: 11: (ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x559fe4c06531]
Jun 18 13:52:23 island sh[7068]: 12: (ThreadPool::WorkThread::entry()+0x10) [0x559fe4c07630]
Jun 18 13:52:23 island sh[7068]: 13: (()+0x76fa) [0x7f4fe256b6fa]
Jun 18 13:52:23 island sh[7068]: 14: (clone()+0x6d) [0x7f4fe05e3b5d]
Jun 18 13:52:23 island sh[7068]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jun 18 13:52:23 island sh[7068]: --- begin dump of recent events ---
Jun 18 13:52:23 island sh[7068]: -2051> 2017-06-18 13:50:36.086036 7f4fe36bb8c0 5 asok(0x559fef2d6000) register_command perfcounters_dump hook 0x559fef216030
/Peter
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
