Hi Benjamin,
honestly the following advice is unlikely to help but you may want to
try to set bluestore_rocksdb_options_annex to one of the following options:
- wal_recovery_mode=kTolerateCorruptedTailRecords
- wal_recovery_mode=kSkipAnyCorruptedRecord
The indication that the setting is in effect would be the respective
value at the end of following log line:
debug 2022-09-12T17:37:05.574+0000 ffffa8316040 4 rocksdb:
Options.wal_recovery_mode: 2
It should get 0 and 3 respectively.
Hoe this helps,
Igor
On 9/12/2022 9:09 PM, Benjamin Naber wrote:
Hi Everybody,
im struggeling now a couple of days with a degraded cehp cluster.
Its a simple 3 node Cluster with 6 OSD´s, 3 SSD based, 3 HDD based. A couple of days ago one of the nodes crashed. in case of Hardisk failure, i replaces the hard disk and the recovery process started without any issues.
As the node was still recovering the new replaced OSD drive was switched to backfillfull. And this is where the pain stareted. I added another node bought a harddrive and wiped the replacement OSD.
The Cluster then was a 4 node sized cluster with 3 OSD´s for the SSD pool and 4 OSD´s for the HDD pool.
Then i started the recovery process from beginning. Ceph has also started at this point a reassingment of missplaced objects.
Then a power failure to one of the remaining nodes happend and now im stucking with a degraded Cluster and 49 pgs inactive, 3 pgs incomplete.
The OSD Container on the power failure node dindt come up anymore in case of rocksdb error. Any advice how the recover the corrupt rocksdb ?
Container Log and rocksdb error:
https://pastebin.com/gvGJdubx
Regards an thanks for your help!
Ben
--
___________________________________________________
Diese E-mail einschließlich eventuell angehängter Dateien enthält vertrauliche und / oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind und diese E-mail irrtümlich erhalten haben, dürfen Sie weder den Inhalt dieser E-mail nutzen noch dürfen Sie die eventuell angehängten Dateien öffnen und auch keine Kopie fertigen oder den Inhalt weitergeben / verbreiten. Bitte verständigen Sie den Absender und löschen Sie diese E-mail und eventuell angehängte Dateien umgehend.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
