Hello good people of linux-raid.
I come here seeking advice on how to repair my raid 5 which I have
managed to royally break.
The raid was initially created in 2009 with 5 1.5TB hdds. Since then I
have replaced 2 drives, 1 with a 2TB and 1 with a 3TB.
The current mdadm version is mdadm - v3.3 - 3rd September 2013
The /etc/debian_version file reads: jessie/sid
The story:
My intention was to replace all the 1.5TB drives with 3TB drives, and
then grow the size of the array.
Yesterday I replaced /dev/sdf, with a 3TB and re-sync'd the array.
The process finished sometime during the night with no errors, so this
morning I continued and replaced /dev/sdb. However, after booting the
machine I was unable to re-mount the partition (the error was on the
lines of: specify filesystem type).
After fumbling around with a couple of mdadm --assembles, removing the
new drive and replacing the old 1.5TB, re-trying some more assembles,
I stumbled upon the wiki article
(https://raid.wiki.kernel.org/index.php/RAID_Recovery) which led me
here.
I thus turn to the list seeking advice/help or simply guidance.
Attached is the current mdadm --examine of all drives, however, my
attempted assembles have clearly damaged something.
Heres the breakdown of the drives currently in the system (90% sure
this is correct):
/dev/sda: OS Hdd, not part of the raid
/dev/sdb: 1.5TB HDD removed this morning
/dev/sdc: 3TB added yesterday, replaced /dev/sdg
/dev/sdd: 2TB HDD, replaced a failed 1.5TB a few years back
/dev/sde: 1.5TB HDD, has not been replaced, should be part of the array
/dev/sdf: 3TB HDD, replaced a failed 1.5TB last year
/dev/sdg: 1.5TB HDD removed yesterday (replaced by /dev/sdc)
My assumption is that I need to try re-creating the array with drives b,c,d,e,f.
Here is the mdadm --assemble --force output for those drives:
mdadm --assemble --force --verbose /dev/md0 /dev/sdb /dev/sdc /dev/sdd
/dev/sde /dev/sdf
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 0.
mdadm: added /dev/sde to /dev/md0 as 1
mdadm: added /dev/sdd to /dev/md0 as 2
mdadm: added /dev/sdb to /dev/md0 as 3 (possibly out of date)
mdadm: no uptodate device for slot 8 of /dev/md0
mdadm: added /dev/sdc to /dev/md0 as 5
mdadm: added /dev/sdf to /dev/md0 as 0
mdadm: /dev/md0 assembled from 3 drives and 1 spare - not enough to
start the array.
Where did this all go wrong:
I believe the 3TB drive that I plugged in this morning came out of a
Seagate BlackArmor NAS400. It thus likely had other superblock
information on it which confused mdadm (purely speculative, I'm not
sure if this is even likely/possible).
I currently have 4 3TB drives spare, however, I do not have any spare
SATA ports to plug them into. While the data on the raid isn't
critical, I would be rather sad (and cry for many days) if it
disappeared.
Any suggestions welcome.
Thanks in advance,
Marnitz Gray
/dev/sda:
MBR Magic : aa55
Partition[0] : 307275192 sectors at 63 (type 83)
Partition[1] : 5301450 sectors at 307275255 (type 05)
/dev/sda2:
MBR Magic : aa55
Partition[0] : 5301387 sectors at 63 (type 82)
/dev/sdb:
Magic : a92b4efc
Version : 0.90.00
UUID : d1413db2:87b7a312:a7ebf782:919a7652
Creation Time : Thu May 7 08:26:38 2009
Raid Level : raid5
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Array Size : 5860553984 (5589.06 GiB 6001.21 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Update Time : Fri May 23 06:24:02 2014
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 1
Spare Devices : 1
Checksum : 30a4c82c - correct
Events : 1167193
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 64 3 active sync /dev/sde
0 0 8 48 0 active sync /dev/sdd
1 1 8 32 1 active sync /dev/sdc
2 2 8 16 2 active sync /dev/sdb
3 3 8 64 3 active sync /dev/sde
4 4 0 0 4 faulty removed
5 5 8 80 5 spare /dev/sdf
/dev/sdc:
Magic : a92b4efc
Version : 0.90.00
UUID : d1413db2:87b7a312:a7ebf782:919a7652
Creation Time : Thu May 7 08:26:38 2009
Raid Level : raid5
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Array Size : 5860553984 (5589.06 GiB 6001.21 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Update Time : Fri May 23 09:41:04 2014
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 2
Spare Devices : 1
Checksum : 30a4f683 - correct
Events : 1167199
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 80 5 spare /dev/sdf
0 0 8 48 0 active sync /dev/sdd
1 1 8 32 1 active sync /dev/sdc
2 2 8 16 2 active sync /dev/sdb
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
5 5 8 80 5 spare /dev/sdf
/dev/sdd:
Magic : a92b4efc
Version : 0.90.00
UUID : d1413db2:87b7a312:a7ebf782:919a7652
Creation Time : Thu May 7 08:26:38 2009
Raid Level : raid5
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Array Size : 5860553984 (5589.06 GiB 6001.21 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Update Time : Fri May 23 09:41:04 2014
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 2
Spare Devices : 1
Checksum : 30a4f643 - correct
Events : 1167199
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 16 2 active sync /dev/sdb
0 0 8 48 0 active sync /dev/sdd
1 1 8 32 1 active sync /dev/sdc
2 2 8 16 2 active sync /dev/sdb
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
5 5 8 80 5 spare /dev/sdf
/dev/sde:
Magic : a92b4efc
Version : 0.90.00
UUID : d1413db2:87b7a312:a7ebf782:919a7652
Creation Time : Thu May 7 08:26:38 2009
Raid Level : raid5
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Array Size : 5860553984 (5589.06 GiB 6001.21 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Update Time : Fri May 23 09:41:04 2014
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 2
Spare Devices : 1
Checksum : 30a4f651 - correct
Events : 1167199
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 32 1 active sync /dev/sdc
0 0 8 48 0 active sync /dev/sdd
1 1 8 32 1 active sync /dev/sdc
2 2 8 16 2 active sync /dev/sdb
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
5 5 8 80 5 spare /dev/sdf
/dev/sdf:
Magic : a92b4efc
Version : 0.90.00
UUID : d1413db2:87b7a312:a7ebf782:919a7652
Creation Time : Thu May 7 08:26:38 2009
Raid Level : raid5
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Array Size : 5860553984 (5589.06 GiB 6001.21 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Update Time : Fri May 23 09:41:04 2014
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 2
Spare Devices : 1
Checksum : 30a4f65f - correct
Events : 1167199
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 48 0 active sync /dev/sdd
0 0 8 48 0 active sync /dev/sdd
1 1 8 32 1 active sync /dev/sdc
2 2 8 16 2 active sync /dev/sdb
3 3 0 0 3 faulty removed
4 4 0 0 4 faulty removed
5 5 8 80 5 spare /dev/sdf
/dev/sdg:
Magic : a92b4efc
Version : 0.90.00
UUID : d1413db2:87b7a312:a7ebf782:919a7652
Creation Time : Thu May 7 08:26:38 2009
Raid Level : raid5
Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
Array Size : 5860553984 (5589.06 GiB 6001.21 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Update Time : Thu May 22 14:07:39 2014
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Checksum : 30a3dfb5 - correct
Events : 1166759
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 16 4 active sync /dev/sdb
0 0 8 64 0 active sync /dev/sde
1 1 8 48 1 active sync /dev/sdd
2 2 8 32 2 active sync /dev/sdc
3 3 8 80 3 active sync /dev/sdf
4 4 8 16 4 active sync /dev/sdb
