Re: A couple OSDs not starting after host reboot

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi All,

from the log output (the line with "Malformed input" string) it rather looks like a device label (the very first 4K data block at main OSD device containing  some basic OSD meta, e.g. OSD UUID) corruption. There are some chances that wrong device has been attached too.

Alison, to investigate further could you please share the 4K superblock content (can be retrieved using dd tool: "dd if=/var/lib/ceph/osd/ceph-665/block count=1 bs=4096 of=./superb.out") and /var/lib/ceph/osd/ceph-665 file listing.


Thanks,

Igor

On 8/25/2023 8:58 PM, Eugen Block wrote:
Hi,
one thing coming to mind is maybe the device names have changed from /dev/sdX to /dev/sdY? Something like that has been reported a couple of times in the last months.

Zitat von Alison Peisker <apeisker@xxxxxxxx>:

Hi all,

We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm.

The error message from the OSD log is ceph-osd[87340]:  ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory

The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600  2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input

We tried restarting the daemons and rebooting the node again, but still see the same error.
Has anyone experienced this issue before? How do we fix this?

Thanks,
Alison
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux