On Fri Oct 28, 2016 at 01:22:31PM +0100, Alexander Shenkin wrote:
> Thanks Andreas, much appreciated. Your points about selftests and smart
> are well taken, and i'll implement them once i get this back up. I'll
> buy yet another new, non drive-from-hell (yes Roman, I did buy the same
> damn drive again. Will try to return it, thanks for the heads up...)
> and follow your instructions below.
>
> One remaining question: is sdc definitely toast? Or, is it possible
> that the Timeout Mismatch (as mentioned by Robin Hill; thanks Robin) is
> flagging the drive as failed, when something else is at play and perhaps
> the drive is actually fine?
>
It's not definitely toast, no (but this is unrelated to the Timeout
mismatches). It has some pending reallocations, which means the drive
was unable to read from some blocks - if a write to the blocks fails
then one of the spare blocks will be reallocated instead, but a write
will often succeed and the pending reallocation will just be cleared.
Unfortunately, reconstruction of the array depends on this data being
readable, so the fact the drive isn't toast doesn't necessarily help.
I'd suggest replicating (using ddrescue) that drive to the new one (when
it arrives) as a first step. It's possible ddrescue will manage to read
the data (it'll make several attempts, so can sometimes read data that
fails initially), otherwise you'll end up with some missing data
(possibly corrupt files, possibly corrupt filesystem metadata, possibly
just a bit of extra noise in an audio/video file). Once that's done, you
can do a proper check on sdc (e.g. a badblocks read/write test), which
will either lead to sector actually being reallocated, or to clearing
the pending reallocations. Unless you get a lot more reallocated sectors
than are currently pending, you can put the drive back into use if you
like (bearing in mind the reputation of these drives and weighing the
replacement cost against the value of your data).
If you run a regular selftest on the array, these sort of issues would
be picked up and repaired automatically (the read errors will trigger
rewrites and either reallocate blocks, clear the pending reallocations,
or fail the drive). Otherwise they're liable to come back to bite you
when you're trying to recover from a different failure.
Timeout Mismatches will lead to drives being failed from an otherwise
healthy array - a read failure on the drive can't be corrected as the
drive is still busy trying when the write request goes through, so the
drive gets kicked out of the array. You didn't say what the issue was
with your original sdb, but if it wasn't a definite fault then it may
have been affected by a timeout mismatch.
Cheers,
Robin
> To everyone: sorry for the multiple posts. Was having majordomo issues...
>
> On 10/27/2016 5:04 PM, Andreas Klauer wrote:
> > On Thu, Oct 27, 2016 at 04:06:14PM +0100, Alexander Shenkin wrote:
> >> md2: raid5 mounted on /, via sd[abcd]3
> >
> > Two failed disks...
> >
> >> md0: raid1 mounted on /boot, via sd[abcd]1
> >
> > Actually only two disks active in that one, the other two are spares.
> > It hardly matters for /boot, but you could grow it to a 4 disk raid1.
> > Spares are not useful.
> >
> >> My sdb was recently reporting problems. Instead of second guessing
> >> those problems, I just got a new disk, replaced it, and added it to
> >> the arrays.
> >
> > Replacing right away is the right thing to do.
> > Unfortunately it seems you have another disk that is broke too.
> >
> >> 2) smartctl (disabled on drives - can enable once back up. should I?)
> >> note: SMART only enabled after problems started cropping up.
> >
> > But... why? Why disable smart? And if you do, is it a surprise that you
> > only notice disk failures when it's already too late?
>
> yeah, i asked myself that same question. there was probably some reason
> I did, but i don't remember what it was. i'll keep smart enabled from
> now on...
>
> > You should enable smart, and not only that, also run regular selftests,
> > and have smartd running, and have it send you mail when something happens.
> > Same with raid checks, raid checks are at least something but it won't
> > tell you about how many reallocated sectors your drive has.
>
> will do
>
> >> root@machinename:/home/username# smartctl --xall /dev/sda
> >
> > Looks fine but never ran a selftest.
> >
> >> root@machinename:/home/username# smartctl --xall /dev/sdb
> >
> > Looks new. (New drives need selftests too.)
> >
> >> root@machinename:/home/username# smartctl --xall /dev/sdc
> >> smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.19.0-39-generic] (local build)
> >> Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
> >>
> >> === START OF INFORMATION SECTION ===
> >> Model Family: Seagate Barracuda 7200.14 (AF)
> >> Device Model: ST3000DM001-1CH166
> >> Serial Number: W1F1N909
> >>
> >> 197 Current_Pending_Sector -O--C- 100 100 000 - 8
> >> 198 Offline_Uncorrectable ----C- 100 100 000 - 8
> >
> > This one is faulty and probably the reason why your resync failed.
> > You have no redundancy left, so an option here would be to get a
> > new drive and ddrescue it over.
> >
> > That's exactly the kind of thing you should be notified instantly
> > about via mail. And it should be discovered when running selftests.
> > Without full surface scan of the media, the disk itself won't know.
> >
> >> ==> WARNING: A firmware update for this drive may be available,
> >> see the following Seagate web pages:
> >> http://knowledge.seagate.com/articles/en_US/FAQ/207931en
> >> http://knowledge.seagate.com/articles/en_US/FAQ/223651en
> >
> > About this, *shrug*
> > I don't have these drives, you might want to check that out.
> > But it probably won't fix bad sectors.
> >
> >> root@machinename:/home/username# smartctl --xall /dev/sdd
> >
> > Some strange things in the error log here, but old.
> > Still, same as for all others - selftest.
> >
> >> ################### mdadm --examine ###########################
> >>
> >> /dev/sda1:
> >> Raid Level : raid1
> >> Raid Devices : 2
> >
> > A RAID 1 with two drives, could be four.
> >
> >> /dev/sdb1:
> >> /dev/sdc1:
> >
> > So these would also have data instead of being spare.
> >
> >> /dev/sda3:
> >> Raid Level : raid5
> >> Raid Devices : 4
> >>
> >> Update Time : Mon Oct 24 09:02:52 2016
> >> Events : 53547
> >>
> >> Device Role : Active device 0
> >> Array State : A..A ('A' == active, '.' == missing)
> >
> > RAID-5 with two failed disks.
> >
> >> /dev/sdc3:
> >> Raid Level : raid5
> >> Raid Devices : 4
> >>
> >> Update Time : Mon Oct 24 08:53:57 2016
> >> Events : 53539
> >>
> >> Device Role : Active device 2
> >> Array State : AAAA ('A' == active, '.' == missing)
> >
> > This one failed, 8:53.
> >
> >> ############ /proc/mdstat ############################################
> >>
> >> md2 : active raid5 sda3[0] sdc3[2](F) sdd3[3]
> >> 8760565248 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2]
> >> [U__U]
> >
> > [U__U] refers to device roles as in [0123],
> > so device role 0 and 3 is okay, 1 and 2 missing.
> >
> >> md0 : active raid1 sdb1[4](S) sdc1[2](S) sda1[0] sdd1[3]
> >> 1950656 blocks super 1.2 [2/2] [UU]
> >
> > Those two spares again, could be [UUUU] instead.
> >
> > tl;dr
> > stop it all,
> > ddrescue /dev/sdc to your new disk,
> > try your luck with --assemble --force (not using /dev/sdc!),
> > get yet another new disk, add, sync, cross fingers.
> >
> > There's also mdadm --replace instead of --remove, --add,
> > that sometimes helps if there's only a few bad sectors
> > on each disk. If the disk you already removed wasn't
> > already kicked from the array by the time you replaced,
> > maybe it would have avoided this problem.
> >
> > But good disk monitoring and testing is even more important.
>
> thanks a bunch, Andreas. I'll monitor and test from now on...
>
> > Regards
> > Andreas Klauer
>
--
___
( ' } | Robin Hill <robin@xxxxxxxxxxxxxxx> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
