Hi, every osd in a SSD that I have upgraded from 15.2.9->15.210 logs
errors like the ones below. The osd's in HD or NVME don't. But they
restart ok and a deep-scrub of the entire pool finishes ok. Could be the
same bug?
2021-04-14T00:29:27.740+0200 7f364750d700 3 rocksdb:
[table/block_based_table_reader.cc:1117] Encountered error while reading
data from compression dictionary block Corruption: truncated block read
from db/044714.sst offset 18446744073709551615, expected 4 bytes, got 0
2021-04-14T00:29:51.852+0200 7f364750d700 3 rocksdb:
[table/block_based_table_reader.cc:1117] Encountered error while reading
data from compression dictionary block Corruption: block checksum
mismatch: expected 0, got 2326482265 in db/044743.sst offset
18446744073709551615 size 18446744073709551615
Br.
El 12/4/21 a las 18:15, Igor Fedotov escribió:
The workaround would be to disable bluestore_fsck_quick_fix_on_mount, do
an upgrade and then do a regular fsck.
Depending on fsck results either proceed with a repair or not.
Thanks,
Igor
On 4/12/2021 6:35 PM, DHilsbos@xxxxxxxxxxxxxx wrote:
Is there a way to check for these zombie blobs, and other issues
needing repair, prior to the upgrade? That would allow us to know
that issues might be coming, and perhaps address them before they
result in corrupt OSDs.
I'm considering upgrading our clusters from 14 to 15, and would really
like to avoid these kinds of issues.
Thank you,
Dominic L. Hilsbos, MBA
Director - Information Technology
Perform Air International Inc.
DHilsbos@xxxxxxxxxxxxxx
www.PerformAir.com
-----Original Message-----
From: Igor Fedotov [mailto:ifedotov@xxxxxxx]
Sent: Monday, April 12, 2021 7:55 AM
To: ceph-users@xxxxxxx
Subject: Re: OSDs RocksDB corrupted when upgrading
nautilus->octopus: unknown WriteBatch tag
Sorry for being too late to the party...
I think the root cause is related to the high amount of repairs made
during the first post-upgrade fsck run.
The check (and fix) for zombie spanning blobs was been backported to
v15.2.9 (here is the PR https://github.com/ceph/ceph/pull/39256). And I
presumt it's the one which causes BlueFS data corruption due to huge
transaction happening during such a repair.
I haven't seen this exact issue (as having that many zombie blobs is a
rarely met bug by itself) but we had to some degree similar issue with
upgrading omap names, see: https://github.com/ceph/ceph/pull/39377
Huge resulting transaction could cause too big write to WAL which in
turn caused data corruption (see https://github.com/ceph/ceph/pull/39701)
Although the fix for the latter has been merged for 15.2.10 some
additional issues with huge transactions might still exist...
If someone can afford another OSD loss it could be interesting to get an
OSD log for such a repair with debug-bluefs set to 20...
I'm planning to make a fix to cap transaction size for repair in the
nearest future anyway though..
Thanks,
Igor
On 4/12/2021 5:15 PM, Dan van der Ster wrote:
Too bad. Let me continue trying to invoke Cunningham's Law for you
... ;)
Have you excluded any possible hardware issues?
15.2.10 has a new option to check for all zero reads; maybe try it
with true?
Option("bluefs_check_for_zeros", Option::TYPE_BOOL,
Option::LEVEL_DEV)
.set_default(false)
.set_flag(Option::FLAG_RUNTIME)
.set_description("Check data read for suspicious pages")
.set_long_description("Looks into data read to check if there is a
4K block entirely filled with zeros. "
"If this happens, we re-read data. If there is
difference, we print error to log.")
.add_see_also("bluestore_retry_disk_reads"),
The "fix zombie spanning blobs" feature was added in 15.2.9. Does
15.2.8 work for you?
Cheers, Dan
On Sun, Apr 11, 2021 at 10:17 PM Jonas Jelten <jelten@xxxxxxxxx> wrote:
Thanks for the idea, I've tried it with 1 thread, and it shredded
another OSD.
I've updated the tracker ticket :)
At least non-racecondition bugs are hopefully easier to spot...
I wouldn't just disable the fsck and upgrade anyway until the cause
is rooted out.
-- Jonas
On 29/03/2021 14.34, Dan van der Ster wrote:
Hi,
Saw that, looks scary!
I have no experience with that particular crash, but I was thinking
that if you have already backfilled the degraded PGs, and can afford
to try another OSD, you could try:
"bluestore_fsck_quick_fix_threads": "1", # because
https://github.com/facebook/rocksdb/issues/5068 showed a similar crash
and the dev said it occurs because WriteBatch is not thread safe.
"bluestore_fsck_quick_fix_on_mount": "false", # should
disable the
fsck during upgrade. See https://github.com/ceph/ceph/pull/40198
-- Dan
On Mon, Mar 29, 2021 at 2:23 PM Jonas Jelten <jelten@xxxxxxxxx> wrote:
Hi!
After upgrading MONs and MGRs successfully, the first OSD host I
upgraded on Ubuntu Bionic from 14.2.16 to 15.2.10
shredded all OSDs on it by corrupting RocksDB, and they now refuse
to boot.
RocksDB complains "Corruption: unknown WriteBatch tag".
The initial crash/corruption occured when the automatic fsck was
ran, and when it committed the changes for a lot of "zombie
spanning blobs".
Tracker issue with logs: https://tracker.ceph.com/issues/50017
Anyone else encountered this error? I've "suspended" the upgrade
for now :)
-- Jonas
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
