Hello,
I have a 10 disk RAID6 array. I had an issue with a molex connector
that caused drives to drop from a 5in3 SATA backplane. There wasn't
any activity occurring on the array during the time, so I think the
filesystem should be in tact. The issue now is the event counts are
out of sync, along with the state of each drive.
# mdadm -A /dev/md1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdk1 /dev/sdl1
/dev/sdm1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1
mdadm: /dev/md1 assembled from 8 drives - not enough to start the
array while not clean - consider --force.
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md1 : inactive sdt1[1](S) sde1[9](S) sdl1[8](S) sdg1[7](S) sdf1[6](S)
sdk1[5](S) sdq1[4](S) sds1[3](S) sdr1[2](S) sdm1[0](S)
19535134720 blocks
# mdadm -Af /dev/md1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdk1 /dev/sdl1
/dev/sdm1 /dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1
mdadm: failed to RUN_ARRAY /dev/md1: Input/output error
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md1 : inactive sdt1[1] sde1[9] sdl1[8] sdg1[7] sdf1[6] sdk1[5] sdq1[4] sdr1[2]
15628107776 blocks
I've read others having success with this but it's not accepting it.
# echo "clean" > /sys/block/md1/md/array_state
bash: echo: write error: Invalid argument
# mdadm -E /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdk1 /dev/sdl1 /dev/sdm1
/dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1 | egrep Event
Events : 147226
Events : 147226
Events : 147226
Events : 147226
Events : 147226
Events : 147221
Events : 147225
Events : 147225
Events : 147221
Events : 147225
# mdadm -E /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdk1 /dev/sdl1 /dev/sdm1
/dev/sdq1 /dev/sdr1 /dev/sds1 /dev/sdt1
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:24:09 2011
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27a7ead3 - correct
Events : 147226
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 9 8 65 9 active sync /dev/sde1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdf1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:24:09 2011
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27a7eadd - correct
Events : 147226
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 81 6 active sync /dev/sdf1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdg1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:24:09 2011
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27a7eaef - correct
Events : 147226
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 8 97 7 active sync /dev/sdg1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdk1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:24:09 2011
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27a7eb2b - correct
Events : 147226
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 161 5 active sync /dev/sdk1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdl1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:24:09 2011
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27a7eb41 - correct
Events : 147226
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 177 8 active sync /dev/sdl1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdm1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 1
Update Time : Wed Feb 9 18:45:59 2011
State : clean
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Checksum : 27aa228c - correct
Events : 147221
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 193 0 active sync /dev/sdm1
0 0 8 193 0 active sync /dev/sdm1
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 65 33 3 active sync /dev/sds1
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdq1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:20:33 2011
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27aa2903 - correct
Events : 147225
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 65 1 4 active sync /dev/sdq1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdr1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:20:33 2011
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27aa290f - correct
Events : 147225
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 65 17 2 active sync /dev/sdr1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sds1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 1
Update Time : Wed Feb 9 18:45:59 2011
State : clean
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Checksum : 27aa222b - correct
Events : 147221
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 65 33 3 active sync /dev/sds1
0 0 8 193 0 active sync /dev/sdm1
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 65 33 3 active sync /dev/sds1
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
/dev/sdt1:
Magic : a92b4efc
Version : 0.90.00
UUID : 0959650a:8a397113:e0647a97:fc2066cc (local to host raid)
Creation Time : Sat Sep 25 13:23:40 2010
Raid Level : raid6
Used Dev Size : 1953513472 (1863.02 GiB 2000.40 GB)
Array Size : 15628107776 (14904.13 GiB 16003.18 GB)
Raid Devices : 10
Total Devices : 8
Preferred Minor : 1
Update Time : Wed Feb 9 19:20:33 2011
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : 27aa292d - correct
Events : 147225
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 65 49 1 active sync /dev/sdt1
0 0 0 0 0 removed
1 1 65 49 1 active sync /dev/sdt1
2 2 65 17 2 active sync /dev/sdr1
3 3 0 0 3 faulty removed
4 4 65 1 4 active sync /dev/sdq1
5 5 8 161 5 active sync /dev/sdk1
6 6 8 81 6 active sync /dev/sdf1
7 7 8 97 7 active sync /dev/sdg1
8 8 8 177 8 active sync /dev/sdl1
9 9 8 65 9 active sync /dev/sde1
I've read others who have had success or failure using create to get
things back in order again. This is what I came up with the drives
listed in sequence from 0 to 9. The drives marked missing are the two
that have event counts much lower than the rest.
# mdadm --create /dev/md1 --verbose --level=6 --raid-devices=10
--chunk=64 missing /dev/sdt1 /dev/sdr1 missing /dev/sdq1 /dev/sdk1
/dev/sdf1 /dev/sdg1 /dev/sdl1 /dev/sde1
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdt1 appears to contain an ext2fs file system
size=-1551761408K mtime=Wed Feb 9 18:45:59 2011
mdadm: /dev/sdt1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdr1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdq1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdk1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdf1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdg1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdl1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: layout defaults to left-symmetric
mdadm: /dev/sde1 appears to contain an ext2fs file system
size=-591265728K mtime=Mon Feb 14 13:17:27 2028
mdadm: /dev/sde1 appears to be part of a raid array:
level=raid6 devices=10 ctime=Sat Sep 25 13:23:40 2010
mdadm: size set to 1953512384K
Continue creating array? n
I haven't executed this yet though.
Any help would be appreciated,
Thanks
ÿôèº{.nÇ+?·?®??+%?Ëÿ±éݶ¥?wÿº{.nÇ+?·¥?{±þ¶¢wø§¶?¡Ü¨}©?²Æ zÚ&j:+v?¨þø ¯ù ®w¥þ?à2?Þ?¨èÚ&¢)ß¡«a¶Úÿÿûàz¿äz¹Þ?ú+?ù???Ý¢jÿ?wèþf
