OSD Crash in recovery: SST file contains data beyond the point of corruption.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Everybody,

im struggeling now a couple of days with a degraded cehp cluster.
Its a simple 3 node Cluster with 6 OSD´s, 3 SSD based, 3 HDD based. A couple of days ago one of the nodes crashed. in case of Hardisk failure, i replaces the hard disk and the recovery process started without any issues.
As the node was still recovering the new replaced OSD drive was switched to backfillfull. And this is where the pain stareted. I added another node bought a harddrive and wiped the replacement OSD.
The Cluster then was a 4 node sized cluster with 3 OSD´s for the SSD pool and 4 OSD´s for the HDD pool.
Then i started the recovery process from beginning. Ceph has also started at this point a reassingment of missplaced objects.
Then a power failure to one of the remaining nodes happend and now im stucking with a degraded Cluster and  49 pgs inactive, 3 pgs incomplete.
The OSD Container on the power failure node dindt come up anymore in case of rocksdb error. Any advice how the recover the corrupt rocksdb ?
Container Log and rocksdb error:

https://pastebin.com/gvGJdubx

Regards an thanks for your help!

Ben


--
___________________________________________________
Diese E-mail einschließlich eventuell angehängter Dateien enthält vertrauliche und / oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind und diese E-mail irrtümlich erhalten haben, dürfen Sie weder den Inhalt dieser E-mail nutzen noch dürfen Sie die eventuell angehängten Dateien öffnen und auch keine Kopie fertigen oder den Inhalt weitergeben / verbreiten. Bitte verständigen Sie den Absender und löschen Sie diese E-mail und eventuell angehängte Dateien umgehend.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux