On 9/16/21 00:09, Davíð Steinn Geirsson wrote:
You might get more information with increasing debug for rocksdb / bluefs /
bluestore
ceph config set osd.0 debug_rocksdb = 20/20
ceph config set osd.0 debug_bluefs = 20/20
ceph config set osd.0 debug_bluestore = 20/20
These debug tuneables give a lot more output and strongly support this being
corrupt RocksDB. The whole OSD output is quite large so I put only the last
part, from the BlueFS journal replay onwards:
https://paste.debian.net/1211916/
I am concerned by this incident, as I know I brought the machine down
cleanly and logs suggest all OSDs were gracefully terminated. Also none
of the drives are reporting any issues with uncorrectable sectors
(though I know not to put too much stock in drive error reporting). In
this case there is sufficient redundancy to correct everything, but if the
same also happened on other hosts that would not be the case. I'll put
the affected drives under a microscope, keep the OSD around for research
just in case, and keep digging hoping to find some explanation.
Yeah, these kind of incidents suck. You think your data is safe, and
than all of a sudden it does not seem to be. Pretty weird. What kind of
drives do you use? I wonder if it can be a cache somewhere that didn't
get flushed (in time).
Thank you very much for your assistance Stefan, it's helped me a lot
getting to know the debugging features of ceph better.
YW.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
