Hi,
I have a server with some 2TB disks, that are partitioned, and those
partitions assembled as RAID1's.
One of the disks has been showing non-zero Current_Pending_Sectors in
smart, so I've added more disks to the machine, partitioned one of the
new disks, and added each of it's partitions to the relevant RAID,
growing the raid to three devices to force the data to be written to the
new disk.
Initially, I did this under single user mode, so that was the only thing
going on on the machine.
One of the old drives (/dev/sda at the time, and the first disk in the
RAID0) then started throwing lots of errors, which seemed to take a long
time to resolve each -- watching this made me think that, under the
circumstances, rather than continuing to read only from /dev/sda, it
might be bright to try reading from /dev/sdb (the other original disk)
in order to provide the data for /dev/sdc (the new disk).
Also, I got the impression that the data on the unreadable blocks was
not being written back to /dev/sda once it was finally read from
/dev/sdb (although confirming that wasn't easy when on the console, with
errors pouring up the screen, and the system being rather unresponsive,
so I rebooted -- after the reboot, it seemed to be getting along better,
so I put it back in production).
After waiting the several days it took to allow the third disk to be
populated with data, I thought I'd try forcing the unreadable sectors to
be written, to get them remapped if they were really bad, or just to get
rid of the Current_Pending_Sector count if it was just a case of the
sectors being corrupt but the physical sector being OK.
[BTW After some rearrangement while I was doing the install, the
doubtful disk is now /dev/sdb, while the newly copied disk is /dev/sdc]
So choosing one of the sectors in question, I did:
root# dd bs=512 skip=19087681 seek=19087681 count=1 if=/dev/sdc of=/dev/sdb
dd: writing `/dev/sdb': Input/output error
1+0 records in
0+0 records out
0 bytes (0 B) copied, 11.3113 s, 0.0 kB/s
Which gives rise to this:
[325487.740650] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[325487.740746] ata2.00: irq_stat 0x00060002, device error via D2H FIS
[325487.740841] ata2.00: failed command: READ DMA
[325487.740924] ata2.00: cmd c8/00:08:40:41:23/00:00:00:00:00/e1 tag 0 dma 4096 in
[325487.740925] res 51/40:00:41:41:23/00:00:01:00:00/e1 Emask 0x9 (media error)
[325487.741153] ata2.00: status: { DRDY ERR }
[325487.741230] ata2.00: error: { UNC }
[325487.749790] ata2.00: configured for UDMA/100
[325487.749797] ata2: EH complete
[325489.757669] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[325489.757759] ata2.00: irq_stat 0x00060002, device error via D2H FIS
[325489.757852] ata2.00: failed command: READ DMA
[325489.757936] ata2.00: cmd c8/00:08:40:41:23/00:00:00:00:00/e1 tag 0 dma 4096 in
[325489.757937] res 51/40:00:41:41:23/00:00:01:00:00/e1 Emask 0x9 (media error)
[325489.758165] ata2.00: status: { DRDY ERR }
[325489.758243] ata2.00: error: { UNC }
[325489.766758] ata2.00: configured for UDMA/100
[325489.766765] ata2: EH complete
[325491.532420] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[325491.532508] ata2.00: irq_stat 0x00060002, device error via D2H FIS
[325491.532595] ata2.00: failed command: READ DMA
[325491.532676] ata2.00: cmd c8/00:08:40:41:23/00:00:00:00:00/e1 tag 0 dma 4096 in
[325491.532677] res 51/40:00:41:41:23/00:00:01:00:00/e1 Emask 0x9 (media error)
[325491.532902] ata2.00: status: { DRDY ERR }
[325491.532978] ata2.00: error: { UNC }
[325491.543346] ata2.00: configured for UDMA/100
[325491.543354] ata2: EH complete
[325493.424305] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[325493.424397] ata2.00: irq_stat 0x00060002, device error via D2H FIS
[325493.424481] ata2.00: failed command: READ DMA
[325493.424571] ata2.00: cmd c8/00:08:40:41:23/00:00:00:00:00/e1 tag 0 dma 4096 in
[325493.424572] res 51/40:00:41:41:23/00:00:01:00:00/e1 Emask 0x9 (media error)
[325493.424792] ata2.00: status: { DRDY ERR }
[325493.424874] ata2.00: error: { UNC }
[325493.435219] ata2.00: configured for UDMA/100
[325493.435226] ata2: EH complete
[325495.190061] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[325495.190145] ata2.00: irq_stat 0x00060002, device error via D2H FIS
[325495.190232] ata2.00: failed command: READ DMA
[325495.190312] ata2.00: cmd c8/00:08:40:41:23/00:00:00:00:00/e1 tag 0 dma 4096 in
[325495.190313] res 51/40:00:41:41:23/00:00:01:00:00/e1 Emask 0x9 (media error)
[325495.190545] ata2.00: status: { DRDY ERR }
[325495.190613] ata2.00: error: { UNC }
[325495.200984] ata2.00: configured for UDMA/100
[325495.200991] ata2: EH complete
[325497.090964] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[325497.091059] ata2.00: irq_stat 0x00060002, device error via D2H FIS
[325497.091160] ata2.00: failed command: READ DMA
[325497.091241] ata2.00: cmd c8/00:08:40:41:23/00:00:00:00:00/e1 tag 0 dma 4096 in
[325497.091242] res 51/40:00:41:41:23/00:00:01:00:00/e1 Emask 0x9 (media error)
[325497.091473] ata2.00: status: { DRDY ERR }
[325497.091552] ata2.00: error: { UNC }
[325497.100074] ata2.00: configured for UDMA/100
[325497.100081] sd 1:0:0:0: [sdb] Unhandled sense code
[325497.100082] sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[325497.100084] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
[325497.100087] Descriptor sense data with sense descriptors (in hex):
[325497.100088] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[325497.100092] 01 23 41 41
[325497.100094] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
[325497.100097] sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 01 23 41 40 00 00 08 00
[325497.100101] end_request: I/O error, dev sdb, sector 19087681
[325497.100197] ata2: EH complete
If I use hdparm's --write-sector on the same sector, it succeeds, and
the dd then succeeds (unless there's another sector following that's
also bad). This doesn't end up resulting in Reallocated_Sector_Ct
increasing (it's still zero on that disk), so it seems that the disk
thinks the physical sector is fine now that it's been written.
I get the impression that for several of the sectors in question,
attempting to write the bad sector revealed a sector one or two
further into the disk that was also corrupt, so despite writing about 20
of them, the Pending sector count has actually gone up from 12 to 32.
Given all that, it seems like this might be a good test case, so I
stopped fixing things in the hope that we'd be able to use the bad
blocks for testing.
I have failed the disk out of the array though (which might be a bit of
an mistake from the testing side of things, but seemed prudent since I'm
serving live data from this server).
So, any suggestions about how I can use this for testing, or why it
appears that mdadm isn't doing it's job a well as it might? I would
think that it should do whatever hdparm's --write-sector does to get the
sector writable again, and then write the data back from the good disk,
since leaving it with the bad blocks means that the RAID is degraded for
those blocks at least.
If it really cannot rewrite the sector then should it not be declaring
the disk faulty? Not that I think that would be the best thing to do in
this circumstance, since it's clearly not _that_ faulty, but blithely
carrying on when some of the data is no longer redundant seems broken as
well.
=-=-=-=-
BTW The machine is running Debian Squeeze, with the kernel being:
linux-image-2.6.35-trunk-amd64_2.6.35-1~experimental.3
The version of mdadm is 3.1.4-1+8efb9d1, hdparm is 9.27-2.1, the drives
in question are all WDC WD2001FASS-00W2B0 (firmware: 01.00101)
Cheers, Phil.
P.S. I'm not subscribed to the list, so please Cc: replies to me.
--
|)| Philip Hands [+44 (0)20 8530 9560] http://www.hands.com/
|-| HANDS.COM Ltd. http://www.uk.debian.org/
|(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND
Attachment:
pgpCkfYRW6g5U.pgp
Description: PGP signature
