I have an OSD process that throws an assert whenever I boot it (see
traceback below).
I have successfully run ceph-bluestore-tool with the commands repair and
fsck, including with the --deep flag, but this did not fix the problem.
Any ideas how to fix this, apart from deleting the whole OSD and
starting over?
Example traceback:
root@d5:~# /usr/bin/ceph-osd -f --cluster ceph --id 16 --setuser ceph
--setgroup ceph
2019-04-15 13:18:21.318 7f9f265a6e00 -1 Public network was set, but
cluster network was not set
2019-04-15 13:18:21.318 7f9f265a6e00 -1 Using public network also
for cluster network
starting osd.16 at - osd_data /var/lib/ceph/osd/ceph-16
/var/lib/ceph/osd/ceph-16/journal
2019-04-15 13:18:48.142 7f9f265a6e00 -1 osd.16 14840 log_to_monitors
{default=true}
/build/ceph-13.2.5/src/osd/PGLog.h: In function 'void
PGLog::reset_complete_to(pg_info_t*)' thread 7f9efba21700 time
2019-04-15 13:18:49.541809
/build/ceph-13.2.5/src/osd/PGLog.h: 777: FAILED assert(log.complete_to
!= log.log.end())
ceph version 13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7f9f1d976c12]
2: (()+0x282dd7) [0x7f9f1d976dd7]
3: (PGLog::reset_complete_to(pg_info_t*)+0x113) [0x55d1409c7c73]
4: (PG::activate(ObjectStore::Transaction&, unsigned int, std::map<int,
std::map<spg_t, pg_query_t, std::less<spg_t>,
std::allocator<std::pair<spg_t const, pg_query_t> > >, std::less<int>,
std::allocator<std::pair<int const, std::map<spg_t, pg_query_t,
std::less<spg_t>, std::allocator<std::pair<spg_t const, pg_query_t> > >
>>> &, std::map<int, std::vector<std::pair<pg_notify_t, PastIntervals>,
std::allocator<std::pair<pg_notify_t, PastIntervals> > >,
std::less<int>, std::allocator<std::pair<int const,
std::vector<std::pair<pg_notify_t, PastIntervals>,
std::allocator<std::pair<pg_notify_t, PastIntervals> > > > > >*,
PG::RecoveryCtx*)+0x2cc) [0x55d14099a52c]
5: (PG::RecoveryState::ReplicaActive::react(PG::Activate const&)+0xbf)
[0x55d14099cccf]
6: (boost::statechart::simple_state<PG::RecoveryState::ReplicaActive,
PG::RecoveryState::Started, PG::RecoveryState::RepNotRecovering,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0x42e) [0x55d1409fbf4e]
7:
(boost::statechart::simple_state<PG::RecoveryState::RepNotRecovering,
PG::RecoveryState::ReplicaActive, boost::mpl::list<mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na>,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0x12e) [0x55d1409f92ae]
8:
(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Initial, std::allocator<void>,
boost::statechart::null_exception_translator>::process_queued_events()+0xb3)
[0x55d1409cefb3]
9:
(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine,
PG::RecoveryState::Initial, std::allocator<void>,
boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base
const&)+0x87) [0x55d1409cf217]
10: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>,
PG::RecoveryCtx*)+0x143) [0x55d1409b50b3]
11: (OSD::dequeue_peering_evt(OSDShard*, PG*,
std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0xcf)
[0x55d1408f59ff]
12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&,
ThreadPool::TPHandle&)+0x50) [0x55d140b63080]
13: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x598) [0x55d140905768]
14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x46e)
[0x7f9f1d97bb4e]
15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f9f1d97dbd0]
16: (()+0x76db) [0x7f9f1c05a6db]
17: (clone()+0x3f) [0x7f9f1b02388f]
Thanks in advance for any help,
Egil
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
