On 9/15/21 21:02, Davíð Steinn Geirsson wrote:
Hi,
On Wed, Sep 15, 2021 at 08:39:11PM +0200, Stefan Kooman wrote:
On 9/15/21 18:06, Davíð Steinn Geirsson wrote:
Just realised the debug paste I sent was for OSD 5 but the other info is for
OSD 0. They are both having the same issue, but for completeness sake here
is the debug output from OSD 0:
http://paste.debian.net/1211873/
All daemons in the cluster are running ceph pacific 16.2.5.
Can you increase debug level for the OSD, i.e. ceph config set osd.0
debug_osd 20/20
And then restart the osd?
Sure, here is the output with 20/20:
https://paste.debian.net/1211886/
Only 3 lines added as far as I can tell:
2021-09-15T18:44:03.289+0000 7fce2827af00 5 object store type is bluestore
[...]
2021-09-15T18:44:05.673+0000 7fce2827af00 2 osd.0 0 init /var/lib/ceph/osd/ceph-0 (looks like hdd)
2021-09-15T18:44:05.673+0000 7fce2827af00 2 osd.0 0 journal /var/lib/ceph/osd/ceph-0/journal
I tried again with debug_osd 99/99 (the maximum) and did not see any
additional messages.
Can you access this link: https://access.redhat.com/solutions/4939871 ?
This would indicate rocksdb corruption. Can you fsck and repair the OSD?
# ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-0/ --debug
# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-0/ --debug
The post states:
If fsck and repair does not help to recover from rocksdb corruption and
all PGs areactive+clean then safest way is to re-deploy affected OSD. In
case some PGs are incomplete or down, Kindly contact Red Hat Ceph Support.
And I agree with that. Better not try to fix things, but let Ceph do a
clean recovery.
As for the Root Cause
The rocksdb corruption on bluestore could be due to hard reboot of OSD
node or block.db device medium error. The rocksdb in blueStore contains
not only OMAPs and metadata for ceph objects, but also ceph objects
layout on disk, entire delayed transactions, allocator free regions and
others.
You might get more information with increasing debug for rocksdb /
bluefs / bluestore
ceph config set osd.0 debug_rocksdb = 20/20
ceph config set osd.0 debug_bluefs = 20/20
ceph config set osd.0 debug_bluestore = 20/20
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
