Hi Ceph users,
after a long time without any major incident (which is great for such a
complex piece of software!), I've finally encountered a problem with our
Ceph installation. All of a sudden the monitor service on one of the
nodes doesn't start anymore. It crashes immediately when I try to start
the service. See below for the journalctl log. I found some
recommendations on the web to check whether the keys/config are all
identical but it seems they are and I actually didn't change anything.
It just stopped working and now it won't run anymore. What can I do to
further diagnose this issue?
Thank you / Best regards
Ranjan
====
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2022-04-12T14:58:49.652+0200
7fa0c67fc640 -1 mon.stage2@-1(???) e3 handle_auth_bad_method hmm, they
didn't like 2 result (13) Permission denied
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2022-04-12T14:58:49.856+0200
7fa0c67fc640 -1 mon.stage2@1(probing) e3 handle_auth_bad_method hmm,
they didn't like 2 result (13) Permission denied
Apr 12 14:58:49 stage2 ceph-mon[614074]: [259B blob data]
Apr 12 14:58:49 stage2 ceph-mon[614074]: PutCF( prefix = mon_sync key =
'latest_monmap' value size = 454)
Apr 12 14:58:49 stage2 ceph-mon[614074]: PutCF( prefix = mon_sync key =
'in_sync' value size = 8)
Apr 12 14:58:49 stage2 ceph-mon[614074]: PutCF( prefix = mon_sync key =
'last_committed_floor' value size = 8)
Apr 12 14:58:49 stage2 ceph-mon[614074]: ./src/mon/MonitorDBStore.h: In
function 'int
MonitorDBStore::apply_transaction(MonitorDBStore::TransactionRef)'
thread 7fa0a1ffb640 time 2022-04-12T>
Apr 12 14:58:49 stage2 ceph-mon[614074]: ./src/mon/MonitorDBStore.h:
351: ceph_abort_msg("failed to write to db")
Apr 12 14:58:49 stage2 ceph-mon[614074]: ceph version 16.2.7
(dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
Apr 12 14:58:49 stage2 ceph-mon[614074]: 1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0xd7)>
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2:
(MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0x890)
[0x5636462d9cc0]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 3:
(Monitor::sync_start(entity_addrvec_t&, bool)+0x370) [0x563646294020]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 4:
(Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0xa2a)
[0x56364629e2ea]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 5:
(Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x2ff)
[0x56364629f8cf]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 6:
(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xe9f)
[0x5636462ace9f]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 7:
(Monitor::_ms_dispatch(Message*)+0x411) [0x5636462ad651]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 8:
(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d)
[0x5636462db5ad]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 9:
(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message>
const&)+0x450) [0x7fa0d0b1d3a0]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 10:
(DispatchQueue::entry()+0x647) [0x7fa0d0b1a757]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 11:
(DispatchQueue::DispatchThread::entry()+0x11) [0x7fa0d0bdb451]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 12:
/lib/x86_64-linux-gnu/libc.so.6(+0x94947) [0x7fa0d0147947]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 13: clone()
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2022-04-12T14:58:49.864+0200
7fa0a1ffb640 -1 ./src/mon/MonitorDBStore.h: In function 'int
MonitorDBStore::apply_transaction(MonitorDBStore::Transact>
Apr 12 14:58:49 stage2 ceph-mon[614074]: ./src/mon/MonitorDBStore.h:
351: ceph_abort_msg("failed to write to db")
Apr 12 14:58:49 stage2 ceph-mon[614074]: ceph version 16.2.7
(dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
Apr 12 14:58:49 stage2 ceph-mon[614074]: 1: (ceph::__ceph_abort(char
const*, int, char const*, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)+0xd7)>
Apr 12 14:58:49 stage2 ceph-mon[614074]: 2:
(MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0x890)
[0x5636462d9cc0]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 3:
(Monitor::sync_start(entity_addrvec_t&, bool)+0x370) [0x563646294020]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 4:
(Monitor::handle_probe_reply(boost::intrusive_ptr<MonOpRequest>)+0xa2a)
[0x56364629e2ea]
Apr 12 14:58:49 stage2 ceph-mon[614074]: 5:
(Monitor::handle_probe(boost::intrusive_ptr<MonOpRequest>)+0x2ff)
[0x56364629f8cf]
===
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
