Greetings,
I have a RAID 6 (which contains a LUKS container) that I'm hoping to get
some help/insight in recovering. The server experienced some sort of
hardware event that resulted in a mandatory restart of the server.
For the record, after the server completed the restart, the array looked
like this, "all spares":
md6 : inactive sdl1[7](S) sdh1[13](S) sdg1[14](S) sdk1[11](S)
sdj1[10](S) sdi1[6](S) sdd1[2](S) sdf1[8](S) sdb1[12](S) sde1[3](S)
sdc1[15](S) 21488638704 blocks super 1.2
The server in question is Ubuntu ("Precise") 12.04.5 LTS with mdadm
version 3.2.5-1ubuntu0 installed.
The mdadm array has the following characteristics:
RAID level: 6
Chunk size: 256k
Version: 1.2
Number of devices: 11
All attempts to assemble the array continued to result in the "all
spare" condition (output above). Thinking that the metadata had been
corrupted somehow, I set out to recreate the array.
The following is the dev_number fields from the metadata, before I
attempted to recreate the array:
# for i in /dev/sd?1; do echo -n $i '' ; dd 2> /dev/null if=$i bs=1
count=4 skip=4256 | od -D | head -n1; done:
/dev/sdb1 0000000 12
/dev/sdc1 0000000 15
/dev/sdd1 0000000 2
/dev/sde1 0000000 3
/dev/sdf1 0000000 8
/dev/sdg1 0000000 14
/dev/sdh1 0000000 13
/dev/sdi1 0000000 6
/dev/sdj1 0000000 10
/dev/sdk1 0000000 11
/dev/sdl1 0000000 7
I used the following to extract the index position of each device on a
device I suspected wasn't corrupted (for the record, they all returned
the same data):
# dd 2> /dev/null if=/dev/sdc1 bs=2 count=6 skip=2176 | od -d
0000000 65534 65534 2 65534 65534 65534 4 5
0000020 65534 65534 7 8 0 9 65534
0000036
As you can see, there's already a visible mismatch between the
dev_number and the listed indexes. For instance, /dev/sdc1 returned a
device number of 15, but there's not a 15th position in the
corresponding list.
I pulled from log history, the last known working "layout", circa July
of this year:
# mdadm -D /dev/md6
/dev/md6:
Version : 1.2
Creation Time : Sat Apr 23 06:22:23 2011
Raid Level : raid6
Array Size : 17581609728 (16767.13 GiB 18003.57 GB)
Used Dev Size : 1953512192 (1863.01 GiB 2000.40 GB)
Raid Devices : 11
Total Devices : 10
Persistence : Superblock is persistent
Update Time : Sat Jun 21 21:13:45 2014
State : clean, degraded
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Name : server:6 (local to host server)
UUID : 6958450b:e4dfa2f3:259ff733:f343a584
Events : 390345
Number Major Minor RaidDevice State
12 8 17 0 active sync /dev/sdb1
3 8 65 1 active sync /dev/sde1
2 8 49 2 active sync /dev/sdd1
8 8 81 3 active sync /dev/sdf1
6 8 129 4 active sync /dev/sdi1
7 8 177 5 active sync /dev/sdl1
6 0 0 6 removed
10 8 145 7 active sync /dev/sdj1
11 8 161 8 active sync /dev/sdk1
13 8 113 9 active sync /dev/sdh1
14 8 97 10 active sync /dev/sdg1
The dev_numbers and index position information in conjunction with the
historic data (directly above) seemed to indicate that the proper
recreation order and command would be the following:
# mdadm --create /dev/md6 --assume-clean --level=6 --raid-devices=11
--metadata=1.2 --chunk=256 /dev/sdb1 /dev/sde1 /dev/sdd1 /dev/sdf1
/dev/sdi1 /dev/sdl1 /dev/sdc1 /dev/sdj1 /dev/sdk1 /dev/sdh1 /dev/sdg1
I ran this above command.
Here is the current output of "lsdrv" which I observed several people
found to be relevant on this list, from creator 'pturmel' on github:
PCI [ahci] 00:11.0 SATA controller: Advanced Micro Devices, Inc.
[AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
âscsi 0:0:0:0 ATA ST31500541AS {6XW03WTD}
ââsda 1.36t [8:0] Partitioned (dos)
â âsda1 109.79m [8:1] ext4 {07f99e8c-95d2-483d-9850-05f04820c3f6}
â ââMounted as /dev/sda1 @ /boot
â âsda2 2.01g [8:2] swap {d137430d-815a-4c45-a394-9bece3aa7136}
â âsda3 7.01g [8:3] ext4 {8db73200-8d9d-4991-9802-b13f1550a9d9}
â ââMounted as /dev/disk/by-uuid/8db73200-8d9d-4991-9802-b13f1550a9d9 @ /
â âsda4 1.36t [8:4] Empty/Unknown
â âdm-0 1.36t [252:0] xfs {db7ddb53-080c-45ba-ab4d-e45d35eb451c}
â âMounted as /dev/mapper/enc @ /encrypted
âscsi 1:0:0:0 ATA ST2000DL003-9VT1 {5YD4VZLV}
ââsdb 1.82t [8:16] Partitioned (gpt)
â âsdb1 1.82t [8:17] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 2:0:0:0 ATA ST2000DM001-1CH1 {Z1E8GNFQ}
ââsdc 1.82t [8:32] Partitioned (gpt)
â âsdc1 1.82t [8:33] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 3:0:0:0 ATA ST2000DL003-9VT1 {5YD2PZM3}
ââsdd 1.82t [8:48] Partitioned (gpt)
â âsdd1 1.82t [8:49] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 4:0:0:0 ATA ST2000DL003-9VT1 {5YD2J0XD}
ââsde 1.82t [8:64] Partitioned (gpt)
â âsde1 1.82t [8:65] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 5:0:0:0 ATA ST2000DL003-9VT1 {5YD3XE9M}
âsdf 1.82t [8:80] Partitioned (gpt)
âsdf1 1.82t [8:81] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
PCI [sata_sil24] 04:04.0 RAID bus controller: Silicon Image, Inc. SiI
3124 PCI-X Serial ATA Controller (rev 02)
âscsi 6:0:0:0 ATA ST2000DL003-9VT1 {5YD6JW2L}
ââsdg 1.82t [8:96] Partitioned (gpt)
â âsdg1 1.82t [8:97] MD raid6 (11) inactive 'server:6'
{65daae65-118b-896a-6205-0f2c4dacb4de}
âscsi 7:0:0:0 ATA ST2000DL003-9VT1 {6YD05E5Y}
ââsdh 1.82t [8:112] Partitioned (gpt)
â âsdh1 1.82t [8:113] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 8:x:x:x [Empty]
âscsi 9:x:x:x [Empty]
PCI [sata_sil24] 06:04.0 RAID bus controller: Silicon Image, Inc. SiI
3124 PCI-X Serial ATA Controller (rev 02)
âscsi 10:0:0:0 ATA ST32000542AS {5XW1PVCZ}
ââsdi 1.82t [8:128] Partitioned (gpt)
â âsdi1 1.82t [8:129] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 11:0:0:0 ATA ST2000DL003-9VT1 {5YD2SND2}
ââsdj 1.82t [8:144] Partitioned (gpt)
â âsdj1 1.82t [8:145] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 12:0:0:0 ATA ST2000DL003-9VT1 {5YD4JTZP}
ââsdk 1.82t [8:160] Partitioned (gpt)
â âsdk1 1.82t [8:161] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
âscsi 13:0:0:0 ATA ST32000542AS {5XW1KAEA}
âsdl 1.82t [8:176] Partitioned (gpt)
âsdl1 1.82t [8:177] MD raid6 (11) inactive 'server:6'
{c88f3b40-c7d4-a33a-5007-ab8ef784c0c2}
Other Block Devices
âloop0 0.00k [7:0] Empty/Unknown
âloop1 0.00k [7:1] Empty/Unknown
âloop2 0.00k [7:2] Empty/Unknown
âloop3 0.00k [7:3] Empty/Unknown
âloop4 0.00k [7:4] Empty/Unknown
âloop5 0.00k [7:5] Empty/Unknown
âloop6 0.00k [7:6] Empty/Unknown
âloop7 0.00k [7:7] Empty/Unknown
âram0 64.00m [1:0] Empty/Unknown
âram1 64.00m [1:1] Empty/Unknown
âram2 64.00m [1:2] Empty/Unknown
âram3 64.00m [1:3] Empty/Unknown
âram4 64.00m [1:4] Empty/Unknown
âram5 64.00m [1:5] Empty/Unknown
âram6 64.00m [1:6] Empty/Unknown
âram7 64.00m [1:7] Empty/Unknown
âram8 64.00m [1:8] Empty/Unknown
âram9 64.00m [1:9] Empty/Unknown
âram10 64.00m [1:10] Empty/Unknown
âram11 64.00m [1:11] Empty/Unknown
âram12 64.00m [1:12] Empty/Unknown
âram13 64.00m [1:13] Empty/Unknown
âram14 64.00m [1:14] Empty/Unknown
âram15 64.00m [1:15] Empty/Unknown
Following the recreation, the array now looks like this:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md6 : active raid6 sdh1[9] sdk1[8] sdj1[7] sdc1[6] sdl1[5] sdi1[4]
sdf1[3] sdd1[2] sde1[1] sdb1[0]
17580439296 blocks super 1.2 level 6, 256k chunk, algorithm 2
[11/10] [UUUUUUUUUUU]
# mdadm -D /dev/md6
/dev/md6:
Version : 1.2
Creation Time : Fri Nov 7 05:40:16 2014
Raid Level : raid6
Array Size : 17580439296 (16766.01 GiB 18002.37 GB)
Used Dev Size : 1953382144 (1862.89 GiB 2000.26 GB)
Raid Devices : 11
Total Devices : 11
Persistence : Superblock is persistent
Update Time : Fri Nov 7 05:40:16 2014
State : clean
Active Devices : 11
Working Devices : 11
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Name : server:6 (local to host server)
UUID : b306872f:5ef902a8:76f5e233:f220f4d4
Events : 0
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 65 1 active sync /dev/sde1
2 8 49 2 active sync /dev/sdd1
3 8 81 3 active sync /dev/sdf1
4 8 129 4 active sync /dev/sdi1
5 8 177 5 active sync /dev/sdl1
6 8 33 6 active sync /dev/sdc1
7 8 145 7 active sync /dev/sdj1
8 8 161 8 active sync /dev/sdk1
9 8 113 9 active sync /dev/sdh1
10 8 97 10 active sync /dev/sdg1
This array should contain a LUKS container, however, it's missing. If I
hexdump the first 20 lines, the LUKS header is completely missing:
# cryptsetup luksOpen /dev/md6 luks
Device /dev/md6 is not a valid LUKS device.
# hexdump -C /dev/md6 | head -n16
00000000 0b 37 89 e0 66 96 7a d4 6c 5b 57 09 a5 8d 6a c5
|.7..f.z.l[W...j.|
00000010 a7 65 20 6e f0 db 74 db 03 d8 e9 2b 39 05 37 a4 |.e
n..t....+9.7.|
00000020 cb 25 d7 7b fd cf b5 b4 12 ad e2 24 24 de 66 42
|.%.{.......$$.fB|
00000030 61 a2 1b ea 8b 5c 04 38 7e 5e 61 11 3d ba 99 35
|a....\.8~^a.=..5|
00000040 b7 e9 e6 76 72 18 d2 d5 bd cd 1b ed 59 15 fb 83
|...vr.......Y...|
00000050 bc 57 94 85 31 c1 3e af 51 f1 25 50 db 57 d3 cd
|.W..1.>.Q.%P.W..|
00000060 69 d5 31 23 df 01 ef 03 e3 92 66 c6 1f 38 3f 57
|i.1#......f..8?W|
00000070 67 20 38 8c c2 ec 25 dc 59 42 b4 5d 9d 9e c1 79 |g
8...%.YB.]...y|
00000080 4a f5 e1 ad f8 08 16 d5 37 3f f6 83 62 f2 6f f5
|J.......7?..b.o.|
00000090 53 95 4f 69 ce 7c ba 4c 86 ef a1 1c 04 d7 b3 17
|S.Oi.|.L........|
000000a0 cd ea 5f 25 56 a4 0d 6f 64 e9 51 b5 71 b3 18 7f
|.._%V..od.Q.q...|
000000b0 46 e7 8b ab 08 ae f5 ed 65 0d 8f 3e 8b 03 25 5c
|F.......e..>..%\|
000000c0 bb 50 dc e6 31 33 4a 88 8e 22 20 72 f0 11 71 d0 |.P..13J.."
r..q.|
000000d0 59 c7 9d 20 f8 e2 f0 f8 75 5f ea 4a 57 d7 d7 9e |Y..
....u_.JW...|
000000e0 c8 05 85 9d d7 cf c9 ab 53 de 11 6f bf d4 e3 b2
|........S..o....|
000000f0 f6 5e 1e 46 5c 16 ae 46 a3 b5 9b f4 b9 ff ca 0c
|.^.F\..F........|
Is the "mdadm --create" operation that I issued, incorrect? Have I done
anything in error?
Unfortunately, I do not have a backup of the LUKS header, as I've
personally never encountered a situation like this, nor was I privy to
the knowledge that LUKS headers should be backed up at all.
If relevant, I did observe that /dev/sdb1 and /dev/sdg1 were showing
"Offline_Uncorrectable" errors via SMART, but nothing that I could
imagine would have contributed to this current predicament.
Is my data gone? Any and all insight are extremly welcomed and appreciated.
Warm regards,
-xar
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
