I have a RAID6 array that has been running nearly trouble-free for years.
This morning, however, the box of disks that contains all the drives for
the array was accidentally switched off while the host computer was still
up and running. I shutdown the host system, turned the JBOD back on, and
then booted the host system back up. Unfortunately, things did not return
to normal.
The array did not auto-assemble at boot, and I have been unable to get it
to auto-assemble once the system is up. I am also not able to manually
assemble the array, and force does not work either. I have worked my way
through the RAID Recovery article in the wiki and I am trying not to panic,
but I've reached the point where the next suggestion is to use --create to
recreate the array (and I'm starting to panic). I'm hoping someone can
offer some help.
When operating normally, my array has 7 disks and 2 spares. Every disk is a
2TB drive. I am using mdadm 3.3.4 on CentOS 6.10.
***My current output from mdadm --detail is:
/dev/md127:
Version : 1.2
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Used Dev Size : 1953510400 (1863.01 GiB 2000.39 GB)
Raid Devices : 7
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Tue Aug 7 13:48:30 2018
State : active, FAILED, Not Started
Active Devices : 0
Working Devices : 2
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 512K
Number Major Minor RaidDevice State
0 0 0 0 removed
2 0 0 2 removed
4 0 0 4 removed
6 0 0 6 removed
8 0 0 8 removed
10 0 0 10 removed
12 0 0 12 removed
8 8 17 - spare /dev/sdb1
13 8 97 - spare /dev/sdg1
***My current output with mdadm --examine for each drive that should be in
the array is:
/dev/sda:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907024002 sectors at 63 (type fd)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 3907024002 sectors at 63 (type fd)
/dev/sde:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)
/dev/sdf:
MBR Magic : aa55
Partition[0] : 3907024002 sectors at 63 (type fd)
/dev/sdg:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)
/dev/sdh:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)
/dev/sdi:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)
***When I boot the system, I note the following relevant entries in my
dmesg:
md: bind<sde1>
md: bind<sdh1>
md: bind<sdd1>
md: bind<sdb1>
md: bind<sdf1>
md: bind<sdg1>
md: bind<sdi1>
md: bind<sda1>
md: bind<sdc1>
md: kicking non-fresh sdc1 from array!
md: unbind<sdc1>
md: export_rdev(sdc1)
md: kicking non-fresh sda1 from array!
md: unbind<sda1>
md: export_rdev(sda1)
md: kicking non-fresh sdi1 from array!
md: unbind<sdi1>
md: export_rdev(sdi1)
md: kicking non-fresh sdf1 from array!
md: unbind<sdf1>
md: export_rdev(sdf1)
md: kicking non-fresh sdd1 from array!
md: unbind<sdd1>
md: export_rdev(sdd1)
md: kicking non-fresh sdh1 from array!
md: unbind<sdh1>
md: export_rdev(sdh1)
md: kicking non-fresh sde1 from array!
md: unbind<sde1>
md: export_rdev(sde1)
raid6: sse2x1 4218 MB/s
raid6: sse2x2 7273 MB/s
raid6: sse2x4 7648 MB/s
raid6: using algorithm sse2x4 (7648 MB/s)
raid6: using intx1 recovery algorithm
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md/raid:md127: allocated 0kB
md/raid:md127: not enough operational devices (7/7 failed)
RAID conf printout:
--- level:6 rd:7 wd:0
md/raid:md127: failed to run raid set.
md: pers->run() failed ...
***Finally, the full current mdadm --examine for each partition is included
below. The last resort section of the recovery article notes that I should
be able to use this information to recreate the array, but it also
recommends contacting the mailing list. As I have never done this before,
I'm definitely nervous to do it on my own. Any guidance would be extremely
appreciated!
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=4096 sectors
State : clean
Device UUID : 66c37bfc:cd0fbe59:c3957f17:54937990
Update Time : Tue Aug 7 13:43:37 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : eb3a10ca - correct
Events : 110575
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=4096 sectors
State : clean
Device UUID : 2c893b0f:64557fe2:a82795fe:95700bff
Update Time : Tue Aug 7 13:48:30 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : dc528c75 - correct
Events : 110595
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : ....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1154 sectors
State : clean
Device UUID : 74d844b0:6e72b687:fdeaec40:8c31390b
Update Time : Tue Aug 7 13:43:37 2018
Checksum : 71207121 - correct
Events : 110575
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1154 sectors
State : clean
Device UUID : 6c4319e3:5f40aefe:c74379d9:48bad82a
Update Time : Tue Aug 7 13:43:37 2018
Checksum : d3188a92 - correct
Events : 110575
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=4096 sectors
State : clean
Device UUID : 7675e981:beb60d0c:8cb07c5b:f6cbe797
Update Time : Tue Aug 7 13:43:37 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6e62bd34 - correct
Events : 110575
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1154 sectors
State : clean
Device UUID : 6a557756:cabeefd1:43a9b8fc:068562fd
Update Time : Tue Aug 7 13:43:37 2018
Checksum : f814b32 - correct
Events : 110575
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=4096 sectors
State : clean
Device UUID : f802e5bc:bb6ce0bc:5a17592b:da48d547
Update Time : Tue Aug 7 13:48:30 2018
Checksum : d9e2e54b - correct
Events : 110595
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : ....... ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=4096 sectors
State : clean
Device UUID : 2ff65cbd:be100744:4c140851:a2717aa5
Update Time : Tue Aug 7 13:43:37 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e4eda15c - correct
Events : 110575
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 95830b7e:f56e6c41:9d02776c:f1f7b162
Name : REDACTED:0 (local to host REDACTED)
Creation Time : Tue Nov 30 18:13:48 2010
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 9767552000 (9315.06 GiB 10001.97 GB)
Used Dev Size : 3907020800 (1863.01 GiB 2000.39 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=4096 sectors
State : clean
Device UUID : 624e8e69:7faf3e8d:52c91751:b27dea02
Update Time : Tue Aug 7 13:43:37 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 37d65960 - correct
Events : 110575
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
