I've been trying to follow the wiki steps to recover my data, and got
to the point where I need to recreate it I believe but would like
someone to confirm or offer advice:
The controller failed and after reboot, I was left with this (full
mdadm -E at bottom of mail):
cat /proc/mdstat
md2 : inactive sdac[14](S) sdn[15](S) sdaf[7](S) sdg[13](S) sdae[4](S)
sdm[0](S) sdl[1](S) sdi[2](S) sdk[5](S) sdj[12](S) sdh[11](S)
sdf[6](S) sde[9](S) sdd[3](S) sdc[8](S)
117208432680 blocks super 1.2
$ mdadm -E /dev/sd{ac,n,af,g,ae,m,l,i,k,j,h,f,e,d,c} | egrep 'Event|/dev/sd'
/dev/sdac:
Events : 485381
/dev/sdn:
Events : 485381
/dev/sdaf:
Events : 485381
/dev/sdg:
Events : 485381
/dev/sdae:
Events : 485381
/dev/sdm:
Events : 485381
/dev/sdl:
Events : 485381
/dev/sdi:
Events : 485381
/dev/sdk:
Events : 485381
/dev/sdj:
Events : 485381
/dev/sdh:
Events : 485381
/dev/sdf:
Events : 485397
/dev/sde:
Events : 485397
/dev/sdd:
Events : 485397
/dev/sdc:
Events : 485397
$ mdadm --stop /dev/md2
mdadm: stopped /dev/md2
$ mdadm --assemble --readonly /dev/md2
/dev/sd{ac,n,af,g,ae,m,l,i,k,j,h,f,e,d,c} --force --verbose
mdadm: looking for devices for /dev/md2
mdadm: /dev/sdac is identified as a member of /dev/md2, slot 14.
mdadm: /dev/sdn is identified as a member of /dev/md2, slot 10.
mdadm: /dev/sdaf is identified as a member of /dev/md2, slot 7.
mdadm: /dev/sdg is identified as a member of /dev/md2, slot 13.
mdadm: /dev/sdae is identified as a member of /dev/md2, slot 4.
mdadm: /dev/sdm is identified as a member of /dev/md2, slot 0.
mdadm: /dev/sdl is identified as a member of /dev/md2, slot 1.
mdadm: /dev/sdi is identified as a member of /dev/md2, slot 2.
mdadm: /dev/sdk is identified as a member of /dev/md2, slot 5.
mdadm: /dev/sdj is identified as a member of /dev/md2, slot 12.
mdadm: /dev/sdh is identified as a member of /dev/md2, slot 11.
mdadm: /dev/sdf is identified as a member of /dev/md2, slot 6.
mdadm: /dev/sde is identified as a member of /dev/md2, slot 9.
mdadm: /dev/sdd is identified as a member of /dev/md2, slot 3.
mdadm: /dev/sdc is identified as a member of /dev/md2, slot 8.
mdadm: added /dev/sdl to /dev/md2 as 1
mdadm: added /dev/sdi to /dev/md2 as 2
mdadm: added /dev/sdd to /dev/md2 as 3
mdadm: added /dev/sdae to /dev/md2 as 4
mdadm: added /dev/sdk to /dev/md2 as 5
mdadm: added /dev/sdf to /dev/md2 as 6
mdadm: added /dev/sdaf to /dev/md2 as 7
mdadm: added /dev/sdc to /dev/md2 as 8
mdadm: added /dev/sde to /dev/md2 as 9
mdadm: added /dev/sdn to /dev/md2 as 10 (possibly out of date)
mdadm: added /dev/sdh to /dev/md2 as 11 (possibly out of date)
mdadm: added /dev/sdj to /dev/md2 as 12 (possibly out of date)
mdadm: added /dev/sdg to /dev/md2 as 13 (possibly out of date)
mdadm: added /dev/sdac to /dev/md2 as 14 (possibly out of date)
mdadm: added /dev/sdm to /dev/md2 as 0
mdadm: /dev/md2 assembled from 10 drives - not enough to start the array.
$ mdadm -E /dev/sd{ac,n,af,g,ae,m,l,i,k,j,h,f,e,d,c}
/dev/sdac:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : b9291948:eb6ede73:54bfd471:5c0299f9
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 8de0f7a3 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 14
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdn:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 8f501e76:359b4a81:e3ea7a3f:ea6d722b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c8d1e1e0 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 10
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdaf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : e76a91fa:2ec466ba:65324998:93adee7d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 31abac56 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdg:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 71af17b2:ff0a9768:36ab7eba:89ca9a78
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b443cd7d - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 13
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdae:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : cd41d218:9628d718:dddaa49e:0fa83d37
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6e078a93 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdm:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 15844233:f0bab6e3:aca369e3:b84c2cb2
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 130accab - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdl:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 6a46b71e:ec247f1a:6605744a:acc74fd1
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : bb75d5a9 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdi:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 8a1cfae6:927ff6df:f4c985e2:1cb603cb
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : daf5b970 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdk:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : ef38d93f:d786ca24:d89d09f4:cf6be53d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : fd0e66b2 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdj:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 4b63410f:cbb2a54f:48c28512:9fd1504f
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 27394749 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdh:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 7c8e9518:c3fd1509:ef0bc6e6:77a8e2fb
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:37:56 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6acfddf1 - correct
Events : 485381
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : b62afdad:88ece794:9e43db9b:792e5d16
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:38:25 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 5b6326a5 - correct
Events : 485397
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : ...A..A.AA..... ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 5c7e81a1:36184ec2:024e8472:d9dfca41
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:38:25 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7e6461c0 - correct
Events : 485397
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : ...A..A.AA..... ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : f8077d3b:a1403a6e:cb88d73d:c67dcf6b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:38:25 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b9a3ec76 - correct
Events : 485397
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : ...A..A.AA..... ('A' == active, '.' == missing, 'R'
== replacing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : bafd5cd8:094b320a:a58f4e67:db4604a3
Name : node-dc4-02:2 (local to host node-dc4-02)
Creation Time : Mon Jun 26 16:10:37 2017
Raid Level : raid6
Raid Devices : 15
Avail Dev Size : 15627791024 (7451.91 GiB 8001.43 GB)
Array Size : 101580637184 (96874.85 GiB 104018.57 GB)
Used Dev Size : 15627790336 (7451.91 GiB 8001.43 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=688 sectors
State : clean
Device UUID : 06a7b126:ae5f52df:b4d8780d:4a71f788
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 12 17:38:25 2018
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2b9ee04 - correct
Events : 485397
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : ...A..A.AA..... ('A' == active, '.' == missing, 'R'
== replacing)
