It's *repeatedly* crashing and restarting? I think the other times we've seen this it was entirely ephemeral and went away on restart, and I really don't know what about this state *could* be made persistent, so that's quite strange. If you can set "debug monc = 20", reproduce this, and post the log it might help us track down the issue.
But if you just want it to work, I bet that restarting the host node will resolve it...
-Greg
On Sat, Jul 14, 2018 at 3:29 PM David Young <david@xxxxxxxxxxxxxxx> wrote:
Hey folks,_______________________________________________
Sorry, posting this from a second account, since for some reason my primary account doesn't seem to be able to post to the list...
I have a Luminous 12.2.6 cluster which suffered a power failure recently. On recovery, one of my OSDs is continually crashing and restarting, with the error below:
----
9ae00 con 0
-3> 2018-07-15 09:50:58.313242 7f131c5a9700 10 monclient: tick
-2> 2018-07-15 09:50:58.313277 7f131c5a9700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-07-15 09:50:28.313274)
-1> 2018-07-15 09:50:58.313320 7f131c5a9700 10 log_client log_queue is 8 last_log 10 sent 0 num 8 unsent 10 sending 10
0> 2018-07-15 09:50:58.320255 7f131c5a9700 -1 /build/ceph-12.2.6/src/common/LogClient.cc: In function 'Message* LogClient::_get_mon_log_message()' thread 7f131c5a9700 time 2018-07-15 09:50:58.313336
/build/ceph-12.2.6/src/common/LogClient.cc: 294: FAILED assert(num_unsent <= log_queue.size())
----
I've found a few recent references to this "FAILED assert" message (assuming that's the cause of the problem), such as https://bugzilla.redhat.com/show_bug.cgi?id=1599718 and http://tracker.ceph.com/issues/18209, with the most recent occurance being 3 days ago (http://tracker.ceph.com/issues/18209#note-12).
Is there any resolution to this issue, or anything I can attempt to recover?
Thanks!
D
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
