On 2018-12-07 01:13, Adam Goryachev wrote: > Wouldn't a mdadm replace solve this? > > https://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array > > The system will copy all readable blocks from |sdd1| to |sdc1|. If it comes to an unreadable block, it will reconstruct it from parity. Once the operation is complete, the former spare (here: |sdc1|) will become active, and the failing drive will be marked as failed (F) so you can remove it. > > Which sounds exactly what you want to do... I have done this now, but it doesn't seem to work as expected. I added the new device and did: # mdadm /dev/md0 --replace /dev/sda1 --with /dev/sdc1 I have in my dmesg now: [ 2338.816160] md/raid1:md0: Disk failure on sda1, disabling device. md/raid1:md0: Operation continuing on 1 devices. [ 2338.816439] ata1: EH complete [ 2345.826094] ata1.00: exception Emask 0x0 SAct 0x1c00000 SErr 0x0 action 0x0 [ 2345.826267] ata1.00: irq_stat 0x40000008 [ 2345.826361] ata1.00: failed command: WRITE FPDMA QUEUED [ 2345.826487] ata1.00: cmd 61/00:b0:d0:30:f1/0a:00:05:00:00/40 tag 22 ncq 1310720 out res 41/10:00:d0:30:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2345.826857] ata1.00: status: { DRDY ERR } [ 2345.826952] ata1.00: error: { IDNF } [ 2345.828243] ata1.00: configured for UDMA/133 [ 2345.828258] ata1: EH complete [ 2352.838317] ata1.00: exception Emask 0x0 SAct 0x70 SErr 0x0 action 0x0 [ 2352.838480] ata1.00: irq_stat 0x40000008 [ 2352.838574] ata1.00: failed command: WRITE FPDMA QUEUED [ 2352.838702] ata1.00: cmd 61/08:20:10:40:06/00:00:00:00:00/40 tag 4 ncq 4096 out res 41/10:00:10:40:06/00:00:00:00:00/40 Emask 0x481 (invalid argument) <F> [ 2352.839065] ata1.00: status: { DRDY ERR } [ 2352.839161] ata1.00: error: { IDNF } [ 2352.840450] ata1.00: configured for UDMA/133 [ 2352.840463] ata1: EH complete [ 2359.850546] ata1.00: exception Emask 0x0 SAct 0xe0000 SErr 0x0 action 0x0 [ 2359.850716] ata1.00: irq_stat 0x40000008 [ 2359.850812] ata1.00: failed command: WRITE FPDMA QUEUED [ 2359.850942] ata1.00: cmd 61/00:88:d0:30:f1/0a:00:05:00:00/40 tag 17 ncq 1310720 out res 41/10:00:d0:30:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2359.851317] ata1.00: status: { DRDY ERR } [ 2359.851414] ata1.00: error: { IDNF } [ 2359.852684] ata1.00: configured for UDMA/133 [ 2359.852698] ata1: EH complete [ 2366.862763] ata1.00: exception Emask 0x0 SAct 0x40000003 SErr 0x0 action 0x0 [ 2366.862939] ata1.00: irq_stat 0x40000008 [ 2366.863033] ata1.00: failed command: WRITE FPDMA QUEUED [ 2366.863159] ata1.00: cmd 61/08:f0:10:40:06/00:00:00:00:00/40 tag 30 ncq 4096 out res 41/10:00:10:40:06/00:00:00:00:00/40 Emask 0x481 (invalid argument) <F> [ 2366.863523] ata1.00: status: { DRDY ERR } [ 2366.863618] ata1.00: error: { IDNF } [ 2366.864897] ata1.00: configured for UDMA/133 [ 2366.864912] ata1: EH complete [ 2373.874946] ata1.00: exception Emask 0x0 SAct 0x7000 SErr 0x0 action 0x0 [ 2373.875113] ata1.00: irq_stat 0x40000008 [ 2373.875207] ata1.00: failed command: WRITE FPDMA QUEUED [ 2373.875333] ata1.00: cmd 61/08:60:10:40:06/00:00:00:00:00/40 tag 12 ncq 4096 out res 41/10:00:10:40:06/00:00:00:00:00/40 Emask 0x481 (invalid argument) <F> [ 2373.875697] ata1.00: status: { DRDY ERR } [ 2373.875792] ata1.00: error: { IDNF } [ 2373.877070] ata1.00: configured for UDMA/133 [ 2373.877084] ata1: EH complete [ 2380.887215] ata1.00: exception Emask 0x0 SAct 0xe000000 SErr 0x0 action 0x0 [ 2380.887388] ata1.00: irq_stat 0x40000008 [ 2380.887481] ata1.00: failed command: WRITE FPDMA QUEUED [ 2380.887610] ata1.00: cmd 61/80:c8:50:2b:f1/05:00:05:00:00/40 tag 25 ncq 720896 out res 41/10:00:50:2b:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2380.887988] ata1.00: status: { DRDY ERR } [ 2380.888085] ata1.00: error: { IDNF } [ 2380.890221] ata1.00: configured for UDMA/133 [ 2380.890235] ata1: EH complete [ 2387.899445] ata1.00: exception Emask 0x0 SAct 0x380 SErr 0x0 action 0x0 [ 2387.899610] ata1.00: irq_stat 0x40000008 [ 2387.899703] ata1.00: failed command: WRITE FPDMA QUEUED [ 2387.899829] ata1.00: cmd 61/08:38:10:40:06/00:00:00:00:00/40 tag 7 ncq 4096 out res 41/10:00:10:40:06/00:00:00:00:00/40 Emask 0x481 (invalid argument) <F> [ 2387.900191] ata1.00: status: { DRDY ERR } [ 2387.900286] ata1.00: error: { IDNF } [ 2387.901562] ata1.00: configured for UDMA/133 [ 2387.901589] sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 2387.901593] sd 0:0:0:0: [sda] tag#7 Sense Key : Illegal Request [current] [descriptor] [ 2387.901596] sd 0:0:0:0: [sda] tag#7 Add. Sense: Logical block address out of range [ 2387.901600] sd 0:0:0:0: [sda] tag#7 CDB: Write(16) 8a 00 00 00 00 00 00 06 40 10 00 00 00 08 00 00 [ 2387.901602] blk_update_request: I/O error, dev sda, sector 409616 [ 2387.901744] md: super_written gets error=-5 [ 2387.901759] ata1: EH complete [ 2394.911896] ata1.00: exception Emask 0x0 SAct 0x600000 SErr 0x0 action 0x0 [ 2394.912063] ata1.00: irq_stat 0x40000008 [ 2394.912157] ata1.00: failed command: WRITE FPDMA QUEUED [ 2394.912287] ata1.00: cmd 61/80:a8:50:2b:f1/05:00:05:00:00/40 tag 21 ncq 720896 out res 41/10:00:50:2b:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2394.912657] ata1.00: status: { DRDY ERR } [ 2394.912752] ata1.00: error: { IDNF } [ 2394.914095] ata1.00: configured for UDMA/133 [ 2394.914109] ata1: EH complete [ 2401.923860] ata1.00: exception Emask 0x0 SAct 0xc SErr 0x0 action 0x0 [ 2401.924020] ata1.00: irq_stat 0x40000008 [ 2401.924114] ata1.00: failed command: WRITE FPDMA QUEUED [ 2401.924244] ata1.00: cmd 61/00:10:d0:30:f1/0a:00:05:00:00/40 tag 2 ncq 1310720 out res 41/10:00:d0:30:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2401.924616] ata1.00: status: { DRDY ERR } [ 2401.924713] ata1.00: error: { IDNF } [ 2401.925999] ata1.00: configured for UDMA/133 [ 2401.926032] sd 0:0:0:0: [sda] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 2401.926036] sd 0:0:0:0: [sda] tag#2 Sense Key : Illegal Request [current] [descriptor] [ 2401.926040] sd 0:0:0:0: [sda] tag#2 Add. Sense: Logical block address out of range [ 2401.926044] sd 0:0:0:0: [sda] tag#2 CDB: Write(16) 8a 00 00 00 00 00 05 f1 30 d0 00 00 0a 00 00 00 [ 2401.926046] blk_update_request: I/O error, dev sda, sector 99692752 [ 2401.926209] ata1: EH complete [ 2401.926239] md: md0: recovery interrupted. [ 2408.936079] ata1.00: exception Emask 0x0 SAct 0x8000 SErr 0x0 action 0x0 [ 2408.936246] ata1.00: irq_stat 0x40000008 [ 2408.936343] ata1.00: failed command: WRITE FPDMA QUEUED [ 2408.936469] ata1.00: cmd 61/80:78:50:2b:f1/05:00:05:00:00/40 tag 15 ncq 720896 out res 41/10:00:50:2b:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2408.936837] ata1.00: status: { DRDY ERR } [ 2408.936932] ata1.00: error: { IDNF } [ 2408.938231] ata1.00: configured for UDMA/133 [ 2408.938242] ata1: EH complete [ 2415.948295] ata1.00: exception Emask 0x0 SAct 0x4000000 SErr 0x0 action 0x0 [ 2415.948468] ata1.00: irq_stat 0x40000008 [ 2415.948562] ata1.00: failed command: WRITE FPDMA QUEUED [ 2415.948688] ata1.00: cmd 61/80:d0:50:2b:f1/05:00:05:00:00/40 tag 26 ncq 720896 out res 41/10:00:50:2b:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2415.949056] ata1.00: status: { DRDY ERR } [ 2415.949151] ata1.00: error: { IDNF } [ 2415.950423] ata1.00: configured for UDMA/133 [ 2415.950434] ata1: EH complete [ 2422.960640] ata1.00: exception Emask 0x0 SAct 0x40 SErr 0x0 action 0x0 [ 2422.960803] ata1.00: irq_stat 0x40000008 [ 2422.960897] ata1.00: failed command: WRITE FPDMA QUEUED [ 2422.961023] ata1.00: cmd 61/80:30:50:2b:f1/05:00:05:00:00/40 tag 6 ncq 720896 out res 41/10:00:50:2b:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2422.961389] ata1.00: status: { DRDY ERR } [ 2422.961484] ata1.00: error: { IDNF } [ 2422.962784] ata1.00: configured for UDMA/133 [ 2422.962795] ata1: EH complete [ 2429.972743] ata1.00: exception Emask 0x0 SAct 0x20000 SErr 0x0 action 0x0 [ 2429.972904] ata1.00: irq_stat 0x40000008 [ 2429.973001] ata1.00: failed command: WRITE FPDMA QUEUED [ 2429.973130] ata1.00: cmd 61/80:88:50:2b:f1/05:00:05:00:00/40 tag 17 ncq 720896 out res 41/10:00:50:2b:f1/00:00:05:00:00/40 Emask 0x481 (invalid argument) <F> [ 2429.973499] ata1.00: status: { DRDY ERR } [ 2429.973594] ata1.00: error: { IDNF } [ 2429.974859] ata1.00: configured for UDMA/133 [ 2429.974878] sd 0:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 2429.974882] sd 0:0:0:0: [sda] tag#17 Sense Key : Illegal Request [current] [descriptor] [ 2429.974886] sd 0:0:0:0: [sda] tag#17 Add. Sense: Logical block address out of range [ 2429.974891] sd 0:0:0:0: [sda] tag#17 CDB: Write(16) 8a 00 00 00 00 00 05 f1 2b 50 00 00 05 80 00 00 [ 2429.974893] blk_update_request: I/O error, dev sda, sector 99691344 [ 2429.975057] ata1: EH complete [ 2430.409191] RAID1 conf printout: [ 2430.409194] --- wd:1 rd:2 [ 2430.409197] disk 0, wo:1, o:0, dev:sda1 [ 2430.409199] disk 1, wo:0, o:1, dev:sdb1 [ 2430.409201] RAID1 conf printout: [ 2430.409202] --- wd:1 rd:2 [ 2430.409204] disk 0, wo:1, o:0, dev:sda1 [ 2430.409205] disk 1, wo:0, o:1, dev:sdb1 [ 2430.409207] RAID1 conf printout: [ 2430.409208] --- wd:1 rd:2 [ 2430.409209] disk 0, wo:1, o:0, dev:sda1 [ 2430.409211] disk 1, wo:0, o:1, dev:sdb1 [ 2430.424383] RAID1 conf printout: [ 2430.424396] --- wd:1 rd:2 [ 2430.424398] disk 0, wo:1, o:1, dev:sdc1 [ 2430.424400] disk 1, wo:0, o:1, dev:sdb1 [ 2430.424591] md: recovery of RAID array md0 [ 2430.424594] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [ 2430.424596] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery. [ 2430.424611] md: using 128k window, over a total of 2929926592k. [ 2430.424612] md: resuming recovery of md0 from checkpoint. [27979.855331] perf interrupt took too long (2508 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 [29786.488257] ata3.00: exception Emask 0x0 SAct 0x3c00000 SErr 0x0 action 0x0 [29786.488421] ata3.00: irq_stat 0x40000008 [29786.488515] ata3.00: failed command: READ FPDMA QUEUED [29786.488640] ata3.00: cmd 60/80:b0:b8:79:f5/05:00:51:01:00/40 tag 22 ncq 720896 in res 41/40:00:18:7c:f5/00:00:51:01:00/40 Emask 0x409 (media error) <F> [29786.489003] ata3.00: status: { DRDY ERR } [29786.489100] ata3.00: error: { UNC } [29786.490659] ata3.00: configured for UDMA/133 [29786.490687] sd 2:0:0:0: [sdb] tag#22 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [29786.490691] sd 2:0:0:0: [sdb] tag#22 Sense Key : Medium Error [current] [descriptor] [29786.490694] sd 2:0:0:0: [sdb] tag#22 Add. Sense: Unrecovered read error - auto reallocate failed [29786.490698] sd 2:0:0:0: [sdb] tag#22 CDB: Read(16) 88 00 00 00 00 01 51 f5 79 b8 00 00 05 80 00 00 [29786.490701] blk_update_request: I/O error, dev sdb, sector 5670009880 [29786.490866] ata3: EH complete [29793.566580] ata3.00: exception Emask 0x0 SAct 0x50001000 SErr 0x0 action 0x0 [29793.566746] ata3.00: irq_stat 0x40000008 [29793.566841] ata3.00: failed command: READ FPDMA QUEUED [29793.566969] ata3.00: cmd 60/00:f0:38:7f:f5/05:00:51:01:00/40 tag 30 ncq 655360 in res 41/40:00:af:81:f5/00:00:51:01:00/40 Emask 0x409 (media error) <F> [29793.567330] ata3.00: status: { DRDY ERR } [29793.567425] ata3.00: error: { UNC } [29793.568901] ata3.00: configured for UDMA/133 [29793.568940] sd 2:0:0:0: [sdb] tag#30 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [29793.568944] sd 2:0:0:0: [sdb] tag#30 Sense Key : Medium Error [current] [descriptor] [29793.568947] sd 2:0:0:0: [sdb] tag#30 Add. Sense: Unrecovered read error - auto reallocate failed [29793.568951] sd 2:0:0:0: [sdb] tag#30 CDB: Read(16) 88 00 00 00 00 01 51 f5 7f 38 00 00 05 00 00 00 [29793.568953] blk_update_request: I/O error, dev sdb, sector 5670011311 [29793.569112] ata3: EH complete [29800.732799] ata3.00: exception Emask 0x0 SAct 0x20008000 SErr 0x0 action 0x0 [29800.732964] ata3.00: irq_stat 0x40000008 [29800.733058] ata3.00: failed command: READ FPDMA QUEUED [29800.733182] ata3.00: cmd 60/80:78:38:84:f5/00:00:51:01:00/40 tag 15 ncq 65536 in res 41/40:00:98:84:f5/00:00:51:01:00/40 Emask 0x409 (media error) <F> [29800.733536] ata3.00: status: { DRDY ERR } [29800.733631] ata3.00: error: { UNC } [29800.735006] ata3.00: configured for UDMA/133 [29800.735031] sd 2:0:0:0: [sdb] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [29800.735035] sd 2:0:0:0: [sdb] tag#15 Sense Key : Medium Error [current] [descriptor] [29800.735038] sd 2:0:0:0: [sdb] tag#15 Add. Sense: Unrecovered read error - auto reallocate failed [29800.735042] sd 2:0:0:0: [sdb] tag#15 CDB: Read(16) 88 00 00 00 00 01 51 f5 84 38 00 00 00 80 00 00 [29800.735044] blk_update_request: I/O error, dev sdb, sector 5670012056 [29800.735207] ata3: EH complete [29872.552210] ata3.00: exception Emask 0x0 SAct 0x30 SErr 0x0 action 0x0 [29872.552375] ata3.00: irq_stat 0x40000008 [29872.552471] ata3.00: failed command: READ FPDMA QUEUED [29872.552595] ata3.00: cmd 60/00:20:b8:bd:f5/09:00:51:01:00/40 tag 4 ncq 1179648 in res 41/40:00:f8:c3:f5/00:00:51:01:00/40 Emask 0x409 (media error) <F> [29872.552951] ata3.00: status: { DRDY ERR } [29872.553046] ata3.00: error: { UNC } [29872.554478] ata3.00: configured for UDMA/133 [29872.554516] sd 2:0:0:0: [sdb] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [29872.554521] sd 2:0:0:0: [sdb] tag#4 Sense Key : Medium Error [current] [descriptor] [29872.554524] sd 2:0:0:0: [sdb] tag#4 Add. Sense: Unrecovered read error - auto reallocate failed [29872.554528] sd 2:0:0:0: [sdb] tag#4 CDB: Read(16) 88 00 00 00 00 01 51 f5 bd b8 00 00 09 00 00 00 [29872.554530] blk_update_request: I/O error, dev sdb, sector 5670028280 [29872.554691] ata3: EH complete [29895.620853] ata3.00: exception Emask 0x0 SAct 0x600000 SErr 0x0 action 0x0 [29895.621015] ata3.00: irq_stat 0x40000008 [29895.621109] ata3.00: failed command: READ FPDMA QUEUED [29895.621234] ata3.00: cmd 60/00:a8:38:fb:f5/05:00:51:01:00/40 tag 21 ncq 655360 in res 41/40:00:78:ff:f5/00:00:51:01:00/40 Emask 0x409 (media error) <F> [29895.621591] ata3.00: status: { DRDY ERR } [29895.621686] ata3.00: error: { UNC } [29895.623081] ata3.00: configured for UDMA/133 [29895.623117] sd 2:0:0:0: [sdb] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [29895.623121] sd 2:0:0:0: [sdb] tag#21 Sense Key : Medium Error [current] [descriptor] [29895.623125] sd 2:0:0:0: [sdb] tag#21 Add. Sense: Unrecovered read error - auto reallocate failed [29895.623129] sd 2:0:0:0: [sdb] tag#21 CDB: Read(16) 88 00 00 00 00 01 51 f5 fb 38 00 00 05 00 00 00 [29895.623131] blk_update_request: I/O error, dev sdb, sector 5670043512 [29895.623291] ata3: EH complete [29917.155816] ata3.00: exception Emask 0x0 SAct 0x1c000000 SErr 0x0 action 0x0 [29917.155982] ata3.00: irq_stat 0x40000008 [29917.156076] ata3.00: failed command: READ FPDMA QUEUED [29917.156201] ata3.00: cmd 60/00:d0:38:16:f6/08:00:51:01:00/40 tag 26 ncq 1048576 in res 41/40:00:58:1c:f6/00:00:51:01:00/40 Emask 0x409 (media error) <F> [29917.156559] ata3.00: status: { DRDY ERR } [29917.156654] ata3.00: error: { UNC } [29917.158492] ata3.00: configured for UDMA/133 [29917.158532] sd 2:0:0:0: [sdb] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [29917.158536] sd 2:0:0:0: [sdb] tag#26 Sense Key : Medium Error [current] [descriptor] [29917.158539] sd 2:0:0:0: [sdb] tag#26 Add. Sense: Unrecovered read error - auto reallocate failed [29917.158543] sd 2:0:0:0: [sdb] tag#26 CDB: Read(16) 88 00 00 00 00 01 51 f6 16 38 00 00 08 00 00 00 [29917.158546] blk_update_request: I/O error, dev sdb, sector 5670050904 [29917.158707] ata3: EH complete [30088.754337] ata3.00: exception Emask 0x0 SAct 0x6030 SErr 0x0 action 0x0 [30088.754504] ata3.00: irq_stat 0x40000008 [30088.754598] ata3.00: failed command: READ FPDMA QUEUED [30088.754722] ata3.00: cmd 60/80:68:38:72:09/05:00:52:01:00/40 tag 13 ncq 720896 in res 41/40:00:f8:76:09/00:00:52:01:00/40 Emask 0x409 (media error) <F> [30088.755079] ata3.00: status: { DRDY ERR } [30088.755173] ata3.00: error: { UNC } [30088.756368] ata3.00: configured for UDMA/133 [30088.756407] sd 2:0:0:0: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [30088.756411] sd 2:0:0:0: [sdb] tag#13 Sense Key : Medium Error [current] [descriptor] [30088.756414] sd 2:0:0:0: [sdb] tag#13 Add. Sense: Unrecovered read error - auto reallocate failed [30088.756418] sd 2:0:0:0: [sdb] tag#13 CDB: Read(16) 88 00 00 00 00 01 52 09 72 38 00 00 05 80 00 00 [30088.756420] blk_update_request: I/O error, dev sdb, sector 5671319288 [30088.756575] ata3: EH complete [30098.957071] ata3.00: exception Emask 0x0 SAct 0xf00 SErr 0x0 action 0x0 [30098.957228] ata3.00: irq_stat 0x40000008 [30098.957322] ata3.00: failed command: READ FPDMA QUEUED [30098.957446] ata3.00: cmd 60/80:40:38:7e:09/08:00:52:01:00/40 tag 8 ncq 1114112 in res 41/40:00:df:83:09/00:00:52:01:00/40 Emask 0x409 (media error) <F> [30098.957807] ata3.00: status: { DRDY ERR } [30098.957904] ata3.00: error: { UNC } [30098.959353] ata3.00: configured for UDMA/133 [30098.959394] sd 2:0:0:0: [sdb] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [30098.959399] sd 2:0:0:0: [sdb] tag#8 Sense Key : Medium Error [current] [descriptor] [30098.959402] sd 2:0:0:0: [sdb] tag#8 Add. Sense: Unrecovered read error - auto reallocate failed [30098.959406] sd 2:0:0:0: [sdb] tag#8 CDB: Read(16) 88 00 00 00 00 01 52 09 7e 38 00 00 08 80 00 00 [30098.959408] blk_update_request: I/O error, dev sdb, sector 5671322591 [30098.959588] ata3: EH complete [30228.710286] ata3.00: exception Emask 0x0 SAct 0x4000001e SErr 0x0 action 0x0 [30228.710460] ata3.00: irq_stat 0x40000008 [30228.710554] ata3.00: failed command: READ FPDMA QUEUED [30228.710679] ata3.00: cmd 60/00:08:38:fb:31/05:00:52:01:00/40 tag 1 ncq 655360 in res 41/40:00:ef:fe:31/00:00:52:01:00/40 Emask 0x409 (media error) <F> [30228.711035] ata3.00: status: { DRDY ERR } [30228.711130] ata3.00: error: { UNC } [30228.712590] ata3.00: configured for UDMA/133 [30228.712626] sd 2:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [30228.712631] sd 2:0:0:0: [sdb] tag#1 Sense Key : Medium Error [current] [descriptor] [30228.712634] sd 2:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed [30228.712638] sd 2:0:0:0: [sdb] tag#1 CDB: Read(16) 88 00 00 00 00 01 52 31 fb 38 00 00 05 00 00 00 [30228.712640] blk_update_request: I/O error, dev sdb, sector 5673975535 [30228.712820] ata3: EH complete [30361.514454] ata3.00: exception Emask 0x0 SAct 0x38000 SErr 0x0 action 0x0 [30361.514615] ata3.00: irq_stat 0x40000008 [30361.514709] ata3.00: failed command: READ FPDMA QUEUED [30361.514833] ata3.00: cmd 60/00:78:b8:ea:46/05:00:52:01:00/40 tag 15 ncq 655360 in res 41/40:00:8f:ed:46/00:00:52:01:00/40 Emask 0x409 (media error) <F> [30361.515190] ata3.00: status: { DRDY ERR } [30361.515285] ata3.00: error: { UNC } [30361.516723] ata3.00: configured for UDMA/133 [30361.516753] sd 2:0:0:0: [sdb] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [30361.516758] sd 2:0:0:0: [sdb] tag#15 Sense Key : Medium Error [current] [descriptor] [30361.516761] sd 2:0:0:0: [sdb] tag#15 Add. Sense: Unrecovered read error - auto reallocate failed [30361.516765] sd 2:0:0:0: [sdb] tag#15 CDB: Read(16) 88 00 00 00 00 01 52 46 ea b8 00 00 05 00 00 00 [30361.516768] blk_update_request: I/O error, dev sdb, sector 5675347343 [30361.516926] ata3: EH complete [30375.270096] ata3.00: exception Emask 0x0 SAct 0x780000 SErr 0x0 action 0x0 [30375.270258] ata3.00: irq_stat 0x40000008 [30375.270355] ata3.00: failed command: READ FPDMA QUEUED [30375.270482] ata3.00: cmd 60/00:98:b8:35:47/05:00:52:01:00/40 tag 19 ncq 655360 in res 41/40:00:5f:3a:47/00:00:52:01:00/40 Emask 0x409 (media error) <F> [30375.270840] ata3.00: status: { DRDY ERR } [30375.270935] ata3.00: error: { UNC } [30375.272365] ata3.00: configured for UDMA/133 [30375.272402] sd 2:0:0:0: [sdb] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [30375.272406] sd 2:0:0:0: [sdb] tag#19 Sense Key : Medium Error [current] [descriptor] [30375.272409] sd 2:0:0:0: [sdb] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed [30375.272413] sd 2:0:0:0: [sdb] tag#19 CDB: Read(16) 88 00 00 00 00 01 52 47 35 b8 00 00 05 00 00 00 [30375.272416] blk_update_request: I/O error, dev sdb, sector 5675367007 [30375.272577] ata3: EH complete [31826.958254] md: md0: recovery done. [31827.032168] RAID1 conf printout: [31827.032172] --- wd:2 rd:2 [31827.032175] disk 0, wo:0, o:1, dev:sdc1 [31827.032177] disk 1, wo:0, o:1, dev:sdb1 I seem to get various types of errors: * Add. Sense: Logical block address out of range * Add. Sense: Unrecovered read error - auto reallocate failed * failed command: WRITE FPDMA QUEUED Right now, my mdadm shows: Number Major Minor RaidDevice State 2 8 33 0 active sync /dev/sdc1 1 8 17 1 active sync /dev/sdb1 0 8 1 - faulty /dev/sda1 Since I got [31826.958254] md: md0: recovery done. at the end, I had hoped that things actually worked in the end, but starting a new scrub via echo check > /sys/block/md0/md/sync_action which I did just now gets me started with new read errors again: [1368746.430658] md: data-check of RAID array md0 [1368746.430663] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [1368746.430664] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. [1368746.430686] md: using 128k window, over a total of 2929926592k. [1368754.461585] ata3.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0 [1368754.461760] ata3.00: irq_stat 0x40000008 [1368754.461860] ata3.00: failed command: READ FPDMA QUEUED [1368754.461989] ata3.00: cmd 60/80:68:80:b5:1a/00:00:00:00:00/40 tag 13 ncq 65536 in res 41/40:00:b8:b5:1a/00:00:00:00:00/40 Emask 0x409 (media error) <F> [1368754.462351] ata3.00: status: { DRDY ERR } [1368754.462451] ata3.00: error: { UNC } [1368754.463919] ata3.00: configured for UDMA/133 [1368754.463962] sd 2:0:0:0: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [1368754.463966] sd 2:0:0:0: [sdb] tag#13 Sense Key : Medium Error [current] [descriptor] [1368754.463969] sd 2:0:0:0: [sdb] tag#13 Add. Sense: Unrecovered read error - auto reallocate failed [1368754.463974] sd 2:0:0:0: [sdb] tag#13 CDB: Read(16) 88 00 00 00 00 00 00 1a b5 80 00 00 00 80 00 00 [1368754.463976] blk_update_request: I/O error, dev sdb, sector 1750456 [1368754.464139] ata3: EH complete [1368785.714727] ata3.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0 [1368785.714897] ata3.00: irq_stat 0x40000008 [1368785.714996] ata3.00: failed command: READ FPDMA QUEUED [1368785.715124] ata3.00: cmd 60/80:d8:00:6e:3f/00:00:00:00:00/40 tag 27 ncq 65536 in res 41/40:00:20:6e:3f/00:00:00:00:00/40 Emask 0x409 (media error) <F> [1368785.715487] ata3.00: status: { DRDY ERR } [1368785.715586] ata3.00: error: { UNC } [1368785.717046] ata3.00: configured for UDMA/133 [1368785.717092] sd 2:0:0:0: [sdb] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [1368785.717096] sd 2:0:0:0: [sdb] tag#27 Sense Key : Medium Error [current] [descriptor] [1368785.717099] sd 2:0:0:0: [sdb] tag#27 Add. Sense: Unrecovered read error - auto reallocate failed [1368785.717103] sd 2:0:0:0: [sdb] tag#27 CDB: Read(16) 88 00 00 00 00 00 00 3f 6e 00 00 00 00 80 00 00 [1368785.717106] blk_update_request: I/O error, dev sdb, sector 4156960 [1368785.717266] ata3: EH complete [1368838.984628] ata3.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0 [1368838.984797] ata3.00: irq_stat 0x40000008 [1368838.984896] ata3.00: failed command: READ FPDMA QUEUED [1368838.985024] ata3.00: cmd 60/80:38:00:5a:bc/00:00:00:00:00/40 tag 7 ncq 65536 in res 41/40:00:00:5a:bc/00:00:00:00:00/40 Emask 0x409 (media error) <F> [1368838.985393] ata3.00: status: { DRDY ERR } [1368838.985494] ata3.00: error: { UNC } [1368838.986940] ata3.00: configured for UDMA/133 [1368838.986979] sd 2:0:0:0: [sdb] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [1368838.986983] sd 2:0:0:0: [sdb] tag#7 Sense Key : Medium Error [current] [descriptor] [1368838.986986] sd 2:0:0:0: [sdb] tag#7 Add. Sense: Unrecovered read error - auto reallocate failed [1368838.986991] sd 2:0:0:0: [sdb] tag#7 CDB: Read(16) 88 00 00 00 00 00 00 bc 5a 00 00 00 00 80 00 00 [1368838.986993] blk_update_request: I/O error, dev sdb, sector 12343808 [1368838.987159] ata3: EH complete [1369117.477937] ata3.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0 [1369117.478107] ata3.00: irq_stat 0x40000008 [1369117.478205] ata3.00: failed command: READ FPDMA QUEUED [1369117.478333] ata3.00: cmd 60/80:50:d0:72:75/00:00:04:00:00/40 tag 10 ncq 65536 in res 41/40:00:10:73:75/00:00:04:00:00/40 Emask 0x409 (media error) <F> [1369117.478695] ata3.00: status: { DRDY ERR } [1369117.478793] ata3.00: error: { UNC } [1369117.480186] ata3.00: configured for UDMA/133 [1369117.480228] sd 2:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [1369117.480232] sd 2:0:0:0: [sdb] tag#10 Sense Key : Medium Error [current] [descriptor] [1369117.480235] sd 2:0:0:0: [sdb] tag#10 Add. Sense: Unrecovered read error - auto reallocate failed [1369117.480239] sd 2:0:0:0: [sdb] tag#10 CDB: Read(16) 88 00 00 00 00 00 04 75 72 d0 00 00 00 80 00 00 [1369117.480242] blk_update_request: I/O error, dev sdb, sector 74806032 [1369117.480409] ata3: EH complete Did I miss anything? Any recommended steps? Should I put the 3 devices in to 3-device RAID-1 and redo the scrub? Niklas
