Hi,
Some further investigation on the failed OSDs:
1 out of 8 OSDs actually has hardware issue,
[16841006.029332] sd 0:0:10:0: [sdj] tag#96 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s
[16841006.037917] sd 0:0:10:0: [sdj] tag#34 FAILED Result:
hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=2s
[16841006.047558] sd 0:0:10:0: [sdj] tag#96 Sense Key : Medium Error
[current]
[16841006.057647] sd 0:0:10:0: [sdj] tag#34 CDB: Read(16) 88 00 00 00 00
00 00 07 e7 70 00 00 00 10 00 00
[16841006.064693] sd 0:0:10:0: [sdj] tag#96 Add. Sense: Unrecovered read
error
[16841006.073988] blk_update_request: I/O error, dev sdj, sector 518000
op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[16841006.080949] sd 0:0:10:0: [sdj] tag#96 CDB: Read(16) 88 00 00 00 00
00 0b 95 d9 80 00 00 00 08 00 00
smartctl:
Error 23 occurred at disk power-on lifetime: 6105 hours (254 days + 9 hours)
When the command that caused the error occurred, the device was
active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 80 d9 95 0b Error: UNC at LBA = 0x0b95d980 = 194369920
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 10 70 e7 07 40 00 14d+02:46:05.704 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 14d+02:46:05.703 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 14d+02:46:05.703 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 14d+02:46:05.703 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 14d+02:46:05.703 READ FPDMA QUEUED
so, let's say, this might be hw fault, though the drive appears to be
working fine.
But the other 7 show no hw related issues. The HDDs are Seagate Exos
X16, enterprise grade, servers are supermicro SSG-6029P-E1CR24L-AT059
with ECC. There are no cpu or memory errors logged in the past months on
the servers, which have been up for ~200 days. So it's is unlikely HW fault.
Is there something else that could be checked? I have left one OSD
intact, so it can be checked further.
Best regards,
Andrej
On 20/09/2021 17:09, Neha Ojha wrote:
Can we please create a bluestore tracker issue for this
(if one does not exist already), where we can start capturing all the
relevant information needed to debug this? Given that this has been
encountered in previous 16.2.* versions, it doesn't sound like a
regression in 16.2.6 to me, rather an issue in pacific. In any case,
we'll prioritize fixing it.
Thanks,
Neha
On Mon, Sep 20, 2021 at 8:03 AM Andrej Filipcic <andrej.filipcic@xxxxxx> wrote:
On 20/09/2021 16:02, David Orman wrote:
Same question here, for clarity, was this on upgrading to 16.2.6 from
16.2.5? Or upgrading
from some other release?
from 16.2.5. but the OSD services were never restarted after upgrade to
.5, so it could be a leftover of previous issues.
Cheers,
Andrej
On Mon, Sep 20, 2021 at 8:57 AM Sean <sean@xxxxxxxxx> wrote:
I also ran into this with v16. In my case, trying to run a repair totally
exhausted the RAM on the box, and was unable to complete.
After removing/recreating the OSD, I did notice that it has a drastically
smaller OMAP size than the other OSDs. I don’t know if that actually means
anything, but just wanted to mention it in case it does.
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
14 hdd 10.91409 1.00000 11 TiB 3.3 TiB 3.2 TiB 4.6 MiB 5.4 GiB
7.7 TiB 29.81 1.02 34 up osd.14
16 hdd 10.91409 1.00000 11 TiB 3.3 TiB 3.3 TiB 20 KiB 9.4 GiB
7.6 TiB 30.03 1.03 35 up osd.16
~ Sean
On Sep 20, 2021 at 8:27:39 AM, Paul Mezzanini <pfmeec@xxxxxxx> wrote:
I got the exact same error on one of my OSDs when upgrading to 16. I
used it as an exercise on trying to fix a corrupt rocksdb. A spent a few
days of poking with no success. I got mostly tool crashes like you are
seeing with no forward progress.
I eventually just gave up, purged the OSD, did a smart long test on the
drive to be sure and then threw it back in the mix. Been HEALTH OK for
a week now after it finished refilling the drive.
On 9/19/21 10:47 AM, Andrej Filipcic wrote:
2021-09-19T15:47:13.610+0200 7f8bc1f0e700 2 rocksdb:
[db_impl/db_impl_compaction_flush.cc:2344] Waiting after background
compaction error: Corruption: block checksum mismatch: expected
2427092066, got 4051549320 in db/251935.sst offset 18414386 size
4032, Accumulated background error counts: 1
2021-09-19T15:47:13.636+0200 7f8bbacf1700 -1 rocksdb: submit_common
error: Corruption: block checksum mismatch: expected 2427092066, got
4051549320 in db/251935.sst offset 18414386 size 4032 code = 2
Rocksdb transaction:
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@xxxxxx
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-425-7074
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@xxxxxx
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-425-7074
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
