Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking
that if you have already backfilled the degraded PGs, and can afford
to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because
https://github.com/facebook/rocksdb/issues/5068 showed a similar crash
and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should disable the
fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@xxxxxxxxx> wrote:
>
> Hi!
>
> After upgrading MONs and MGRs successfully, the first OSD host I upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10
> shredded all OSDs on it by corrupting RocksDB, and they now refuse to boot.
> RocksDB complains "Corruption: unknown WriteBatch tag".
>
> The initial crash/corruption occured when the automatic fsck was ran, and when it committed the changes for a lot of "zombie spanning blobs".
>
> Tracker issue with logs: https://tracker.ceph.com/issues/50017
>
>
> Anyone else encountered this error? I've "suspended" the upgrade for now :)
>
> -- Jonas
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
