A friend of mine has got a serious problem after replacing a failed disk
in a RAID-6 array.
His setup is:
- 15 SATA drives connected to a LSI 9200 SAS HBA
- Debian 6.0.10 (Squeeze)
- Open Media Vault 0.5.54
The array was created using OMV graphical interface as RAID6.
When the Raid Device number 9 has failed the system was shutdown and the
drive was replaced.
Then the system was started - but the array did not rebuild (as was
expected). It showed up as FAILED with 3 drives marked as "removed".
The current state is:
###############
/dev/md127:
Version : 1.2
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
Raid Devices : 15
Total Devices : 12
Persistence : Superblock is persistent
Update Time : Thu Dec 14 10:45:56 2017
State : active, FAILED, Not Started
Active Devices : 12
Working Devices : 12
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : ebhnas:RAID01
UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Events : 1575050
Number Major Minor RaidDevice State
0 8 144 0 active sync /dev/sdj
1 8 224 1 active sync /dev/sdo
2 8 48 2 active sync /dev/sdd
3 0 0 3 removed
4 8 192 4 active sync /dev/sdm
5 8 208 5 active sync /dev/sdn
6 8 128 6 active sync /dev/sdi
15 8 112 7 active sync /dev/sdh
8 8 160 8 active sync /dev/sdk
9 0 0 9 removed
10 8 176 10 active sync /dev/sdl
11 0 0 11 removed
12 8 32 12 active sync /dev/sdc
13 8 0 13 active sync /dev/sda
14 8 16 14 active sync /dev/sdb
###############
I have checked the events count for all drives. It is:
- 1575050 for all drives marked as "active sync"
- 1573779 for drives "/dev/sdg" and "/dev/sdp" (RaidDevice number 3 and
11, respectively)
- the replaced drive "/dev/sdf" is empty
What would be the safest strategy to try to recover data from this
array? Is it still possible?
I send attached the status report for all the drives in the array
(except for the replaced one).
Any help will be much appreciated.
Ryszard
/dev/sdj:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 8d4ef54a:69989b53:b02d7b94:2267fa4d
Update Time : Thu Dec 14 10:45:56 2017
Checksum : 5f6f0fc0 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdo:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 740cc99a:72e630b0:281f6ae5:fdb2a33e
Update Time : Thu Dec 14 10:45:56 2017
Checksum : 4d705905 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c6e3e409:7b8713fe:36c63ced:eb5b5803
Update Time : Thu Dec 14 10:45:56 2017
Checksum : d6f6215c - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdg:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 0de28475:96c9b4c8:c49ac8f3:f2316925
Update Time : Wed Dec 13 17:21:12 2017
Checksum : 35e52ad8 - correct
Events : 1573779
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAA.AAAAA ('A' == active, '.' == missing)
/dev/sdm:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 5d294233:d48821f7:941ef54a:c219704c
Update Time : Thu Dec 14 10:45:56 2017
Checksum : a0317e83 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdn:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a086cc83:437c5a6e:8e10e80d:a816357d
Update Time : Thu Dec 14 10:45:56 2017
Checksum : 5bacbe16 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdi:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fdbfc998:756addfd:46321783:40bd6cff
Update Time : Thu Dec 14 10:45:56 2017
Checksum : f793adf7 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdh:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 070ab6cb:ef8c30b1:a0fc256d:0228af09
Update Time : Thu Dec 14 10:45:56 2017
Checksum : d224581f - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdk:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1c1d49c8:38146dcf:91fca3de:72353f5c
Update Time : Thu Dec 14 10:45:56 2017
Checksum : b101f758 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdl:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 6ab07d37:10ec76f9:95447668:122c64c3
Update Time : Thu Dec 14 10:45:56 2017
Checksum : 3b37a124 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 10
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdp:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 48eb2c5b:b204e306:9cdef159:ee08aae3
Update Time : Wed Dec 13 17:21:12 2017
Checksum : 7e25a7da - correct
Events : 1573779
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAAAAA.AAAAA ('A' == active, '.' == missing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 33d3f559:6700579f:abe9ffb6:a17a05c0
Update Time : Thu Dec 14 10:45:56 2017
Checksum : 4ebacbeb - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 12
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 86e17f17:544742e0:a8b9568a:4ea1f87f
Update Time : Thu Dec 14 10:45:56 2017
Checksum : e07a17d5 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 13
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Name : ebhnas:RAID01
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 50791336960 (24219.20 GiB 26005.16 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 65214f76:0061e610:fbd0fa8f:cc8cffaf
Update Time : Thu Dec 14 10:45:56 2017
Checksum : a5987432 - correct
Events : 1575050
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 14
Array State : AAA.AAAAA.A.AAA ('A' == active, '.' == missing)
/dev/md127:
Version : 1.2
Creation Time : Thu Dec 12 11:11:55 2013
Raid Level : raid6
Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
Raid Devices : 15
Total Devices : 12
Persistence : Superblock is persistent
Update Time : Thu Dec 14 10:45:56 2017
State : active, FAILED, Not Started
Active Devices : 12
Working Devices : 12
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : ebhnas:RAID01
UUID : ebef4f20:09a906b2:57fd6e29:1da8c9c9
Events : 1575050
Number Major Minor RaidDevice State
0 8 144 0 active sync /dev/sdj
1 8 224 1 active sync /dev/sdo
2 8 48 2 active sync /dev/sdd
3 0 0 3 removed
4 8 192 4 active sync /dev/sdm
5 8 208 5 active sync /dev/sdn
6 8 128 6 active sync /dev/sdi
15 8 112 7 active sync /dev/sdh
8 8 160 8 active sync /dev/sdk
9 0 0 9 removed
10 8 176 10 active sync /dev/sdl
11 0 0 11 removed
12 8 32 12 active sync /dev/sdc
13 8 0 13 active sync /dev/sda
14 8 16 14 active sync /dev/sdb
