Dear list,
a friend has a 4 x 2TB RAID10 array (QNAP NAS) that stopped working after a long power outage. According to "dmesg" and "mdadm --examine" outputs (attached at the end of this email) it seems the event counts are slightly off and the RAID array is unable to start?
Here are some of the things we tried to find and fix the problem, albeit to no avail:
- SMART status of all 4 drives seems to be OK (at least according to QNAP web interface)
- mdadm --assemble /dev/md0 /dev/sd{a,b,c,d}3
- mdadm --force --assemble /dev/md0 /dev/sd{a,b,c,d}3
The above commands fail due to not enough operational drives (I forgot to copy/paste the exact error message and don't have access to the RAID atm).
He'll get 4 new drives to take backup copies of all 4 RAID drives, so we'll probably wait until then before we try anything else.
Is there something obvious that we're missing or should try?
Thank you, any help is greatly appreciated!
[~] # dmesg
[...]
[ 42.950162] md: md0 stopped.
[ 42.958105] md: bind<sdb3>
[ 42.958314] md: bind<sdc3>
[ 42.958516] md: bind<sdd3>
[ 42.958720] md: bind<sda3>
[ 42.958772] md: md0 stopped.
[ 42.958780] md: unbind<sda3>
[ 42.966821] md: export_rdev(sda3)
[ 42.966879] md: unbind<sdd3>
[ 42.974817] md: export_rdev(sdd3)
[ 42.974856] md: unbind<sdc3>
[ 42.981817] md: export_rdev(sdc3)
[ 42.981855] md: unbind<sdb3>
[ 42.989817] md: export_rdev(sdb3)
[ 43.006623] md: md0 stopped.
[ 43.014457] md: bind<sdb3>
[ 43.014665] md: bind<sdc3>
[ 43.014869] md: bind<sdd3>
[ 43.015069] md: bind<sda3>
[ 43.015106] md: kicking non-fresh sdd3 from array!
[ 43.015114] md: unbind<sdd3>
[ 43.023824] md: export_rdev(sdd3)
[ 43.023867] md: kicking non-fresh sdc3 from array!
[ 43.023874] md: unbind<sdc3>
[ 43.030819] md: export_rdev(sdc3)
[ 43.032291] md/raid10:md0: not enough operational mirrors.
[ 43.032331] md: pers->run() failed ...
[ 43.032384] md: md0 stopped.
[ 43.032394] md: unbind<sda3>
[ 43.043822] md: export_rdev(sda3)
[ 43.043836] md: unbind<sdb3>
[ 43.051820] md: export_rdev(sdb3)
[...]
[~] # mdadm --examine /dev/sd{a,b,c,d}3
/dev/sda3:
Magic : a92b4efc
Version : 0.90.00
UUID : ebec3f57:7faefc30:e47ecd26:6fb17990
Creation Time : Tue Sep 4 16:20:29 2012
Raid Level : raid10
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 3903891200 (3723.04 GiB 3997.58 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Tue Nov 27 08:55:49 2018
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Checksum : 899a0111 - correct
Events : 220057
Layout : near=2
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 3 0 active sync /dev/sda3
0 0 8 3 0 active sync /dev/sda3
1 1 8 19 1 active sync /dev/sdb3
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
/dev/sdb3:
Magic : a92b4efc
Version : 0.90.00
UUID : ebec3f57:7faefc30:e47ecd26:6fb17990
Creation Time : Tue Sep 4 16:20:29 2012
Raid Level : raid10
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 3903891200 (3723.04 GiB 3997.58 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Tue Nov 27 08:55:49 2018
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Checksum : 899a0123 - correct
Events : 220057
Layout : near=2
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 19 1 active sync /dev/sdb3
0 0 8 3 0 active sync /dev/sda3
1 1 8 19 1 active sync /dev/sdb3
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
/dev/sdc3:
Magic : a92b4efc
Version : 0.90.00
UUID : ebec3f57:7faefc30:e47ecd26:6fb17990
Creation Time : Tue Sep 4 16:20:29 2012
Raid Level : raid10
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 3903891200 (3723.04 GiB 3997.58 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Tue Nov 27 08:53:42 2018
State : active
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 8996a500 - correct
Events : 220053
Layout : near=2
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 35 2 active sync /dev/sdc3
0 0 8 3 0 active sync /dev/sda3
1 1 8 19 1 active sync /dev/sdb3
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
/dev/sdd3:
Magic : a92b4efc
Version : 0.90.00
UUID : ebec3f57:7faefc30:e47ecd26:6fb17990
Creation Time : Tue Sep 4 16:20:29 2012
Raid Level : raid10
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 3903891200 (3723.04 GiB 3997.58 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Tue Nov 27 08:53:42 2018
State : active
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 8996a512 - correct
Events : 220053
Layout : near=2
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 51 3 active sync /dev/sdd3
0 0 8 3 0 active sync /dev/sda3
1 1 8 19 1 active sync /dev/sdb3
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
