Dan,
Yes, I have had this happen several times since, but fortunately the
last couple of times has only happened to one or two OSDs at a time so
it didn't take down entire pools. Remedy has been the same.
I had been holding off on too much further investigation because I
thought the source of the issue may have been some old hardware
gremlins, and we're waiting on some new hardware.
-Troy
On 2/20/20 1:40 PM, Dan van der Ster wrote:
Hi Troy,
Looks like we hit the same today -- Sage posted some observations
here: https://tracker.ceph.com/issues/39525#note-6
Did it happen again in your cluster?
Cheers, Dan
On Tue, Aug 20, 2019 at 2:18 AM Troy Ablan <tablan@xxxxxxxxx> wrote:
While I'm still unsure how this happened, this is what was done to solve
this.
Started OSD in foreground with debug 10, watched for the most recent
osdmap epoch mentioned before abort(). For example, if it mentioned
that it just tried to load 80896 and then crashed
# ceph osd getmap -o osdmap.80896 80896
# ceph-objectstore-tool --op set-osdmap --data-path
/var/lib/ceph/osd/ceph-77/ --file osdmap.80896
Then I restarted the osd in foreground/debug, and repeated for the next
osdmap epoch until it got past the first few seconds. This process
worked for all but two OSDs. For the ones that succeeded I'd ^C and
then start the osd via systemd
For the remaining two, it would try loading the incremental map and then
crash. I had presence of mind to make dd images of every OSD before
starting this process, so I reverted these two to the state before
injecting the osdmaps.
I then injected the last 15 or so epochs of the osdmap in sequential
order before starting the osd, with success.
This leads me to believe that the step-wise injection didn't work
because the osd had more subtle corruption that it got past, but it was
confused when it requested the next incremental delta.
Thanks again to Brad/badone for the guidance!
Tracker issue updated.
Here's the closing IRC dialogue re this issue (UTC-0700)
2019-08-19 16:27:42 < MooingLemur> badone: I appreciate you reaching out
yesterday, you've helped a ton, twice now :) I'm still concerned
because I don't know how this happened. I'll feel better once
everything's active+clean, but it's all at least active.
2019-08-19 16:30:28 < badone> MooingLemur: I had a quick discussion with
Josh earlier and he shares my opinion this is likely somehow related to
these drives or perhaps controllers, or at least specific to these machines
2019-08-19 16:31:04 < badone> however, there is a possibility you are
seeing some extremely rare race that no one up to this point has seen before
2019-08-19 16:31:20 < badone> that is less likely though
2019-08-19 16:32:50 < badone> the osd read the osdmap over the wire
successfully but wrote it out to disk in a format that it could not then
read back in (unlikely) or...
2019-08-19 16:33:21 < badone> the map "changed" after it had been
written to disk
2019-08-19 16:33:46 < badone> the second is considered most likely by us
but I recognise you may not share that opinion
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
