Hi All,
from the log output (the line with "Malformed input" string) it rather
looks like a device label (the very first 4K data block at main OSD
device containing some basic OSD meta, e.g. OSD UUID) corruption. There
are some chances that wrong device has been attached too.
Alison, to investigate further could you please share the 4K superblock
content (can be retrieved using dd tool: "dd
if=/var/lib/ceph/osd/ceph-665/block count=1 bs=4096 of=./superb.out")
and /var/lib/ceph/osd/ceph-665 file listing.
Thanks,
Igor
On 8/25/2023 8:58 PM, Eugen Block wrote:
Hi,
one thing coming to mind is maybe the device names have changed from
/dev/sdX to /dev/sdY? Something like that has been reported a couple
of times in the last months.
Zitat von Alison Peisker <apeisker@xxxxxxxx>:
Hi all,
We rebooted all the nodes in our 17.2.5 cluster after performing
kernel updates, but 2 of the OSDs on different nodes are not coming
back up. This is a production cluster using cephadm.
The error message from the OSD log is ceph-osd[87340]: ** ERROR:
unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No
such file or directory
The error message from ceph-volume is 2023-08-23T16:12:43.452-0500
7f0cad968600 2
bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50)
_read_bdev_label unable to decode label at offset 102: void
bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)
decode past end of struct encoding: Malformed input
We tried restarting the daemons and rebooting the node again, but
still see the same error.
Has anyone experienced this issue before? How do we fix this?
Thanks,
Alison
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
