Re: rocksdb corruption with 16.2.6

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



 In my case it happened after upgrading from v16.2.4 to v16.2.5 a couple
months ago.

~ Sean


On Sep 20, 2021 at 9:02:45 AM, David Orman <ormandj@xxxxxxxxxxxx> wrote:

> Same question here, for clarity, was this on upgrading to 16.2.6 from
> 16.2.5? Or upgrading
> from some other release?
>
> On Mon, Sep 20, 2021 at 8:57 AM Sean <sean@xxxxxxxxx> wrote:
>
>
>  I also ran into this with v16. In my case, trying to run a repair totally
>
> exhausted the RAM on the box, and was unable to complete.
>
>
> After removing/recreating the OSD, I did notice that it has a drastically
>
>  smaller OMAP size than the other OSDs. I don’t know if that actually means
>
> anything, but just wanted to mention it in case it does.
>
>
> ID   CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE  DATA     OMAP     META
>
>   AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
>
> 14   hdd    10.91409   1.00000   11 TiB  3.3 TiB  3.2 TiB  4.6 MiB  5.4 GiB
>
>  7.7 TiB  29.81  1.02   34      up    osd.14
>
> 16   hdd    10.91409   1.00000   11 TiB  3.3 TiB  3.3 TiB   20 KiB  9.4 GiB
>
>  7.6 TiB  30.03  1.03   35      up    osd.16
>
>
> ~ Sean
>
>
>
> On Sep 20, 2021 at 8:27:39 AM, Paul Mezzanini <pfmeec@xxxxxxx> wrote:
>
>
> > I got the exact same error on one of my OSDs when upgrading to 16.  I
>
> > used it as an exercise on trying to fix a corrupt rocksdb. A spent a few
>
> > days of poking with no success.  I got mostly tool crashes like you are
>
> > seeing with no forward progress.
>
> >
>
> > I eventually just gave up, purged the OSD, did a smart long test on the
>
> > drive to be sure and then threw it back in the mix.  Been HEALTH OK for
>
> > a week now after it finished refilling the drive.
>
> >
>
> >
>
> > On 9/19/21 10:47 AM, Andrej Filipcic wrote:
>
> >
>
> > 2021-09-19T15:47:13.610+0200 7f8bc1f0e700  2 rocksdb:
>
> >
>
> > [db_impl/db_impl_compaction_flush.cc:2344] Waiting after background
>
> >
>
> > compaction error: Corruption: block checksum mismatch: expected
>
> >
>
> > 2427092066, got 4051549320  in db/251935.sst offset 18414386 size
>
> >
>
> > 4032, Accumulated background error counts: 1
>
> >
>
> > 2021-09-19T15:47:13.636+0200 7f8bbacf1700 -1 rocksdb: submit_common
>
> >
>
> > error: Corruption: block checksum mismatch: expected 2427092066, got
>
> >
>
> > 4051549320  in db/251935.sst offset 18414386 size 4032 code = 2
>
> >
>
> > Rocksdb transaction:
>
> >
>
> > _______________________________________________
>
> > ceph-users mailing list -- ceph-users@xxxxxxx
>
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
> >
>
> _______________________________________________
>
> ceph-users mailing list -- ceph-users@xxxxxxx
>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux