Hi Neil,
regrettably, I do not have logs from Jun 9th. This is what happened, in Detail:
Before I grew the RAID, I made a backup of the system drive (Sometime
around the beginning of may). Then I grew the RAID and the dm-crypt
container on it.
I then noticed that ext4 filesystems cannot be grown above a certain
limit, which is why I decided to convert to BTRFS.
Prior to Jun 9th I upgraded Ubuntu from 12.04 LTS to 14.04 LTS. The
reason was that I wanted the newest BTRFS utils for the conversion.
The conversion went smoothly, but the Ubuntu upgrade messed with some
services running on the server (e.g. various configs for web apps,
nothing to do with the raid). So I wanted to do a fresh install. I
didn't do a backup of the system, because I had the old backup which
had worked before.
I attempted the fresh install, looking at the disks with GParted
beforehand (as I said earlier, my theory is that GParted might have
messed up some of the md superblocks).
So after the fresh install, I wasn't able to start the RAID (error
message was input/output error).
So I thought I'll just restore the old backup, since that worked
perfectly, and then make my way from there.
After the restore, The system asked me if I wanted to start a degraded
RAID. I thought it meant the raid was degraded because of the failing
drive, and said yes.
It then showed me a Raid with 6 Drives, all spares. At this point the
panic started to set in :(
I have attached some log excerpts from the beginning of may, before I
made the backup and the old RAID was functioning (kern.log and syslog,
grepped for 'md').
Furthermore, searching for the superblock with od gave me the following:
od -x /dev/sdh | grep '4efc a92b'
20234525260 8a2a c251 a28b 2f92 f63e 8d72 4efc a92b
103362752200 4efc a92b 3412 ad92 b451 bc40 5897 d215
od -x /dev/sdi | grep '4efc a92b'
135674640060 4efc a92b 89de a9d8 d2b8 395e 6f37 4597
I don't think those are the superblocks, but rather the "magic number"
being present somewhere on the drive :(
Doing further research I found this:
http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/
Is there any "safe" way to restore the superblocks, or is re-creating
the RAID my final option?
Thanks again,
-P.
> Well, you've definitely made progress. You've found 6 of the devices.
> They all look consistent and it appears the array was completely coherent at
> Mon Jun 9 21:52:48 2014
>
> You think that the 7th device is dead or dying, so you just need to find 2
> more (1 would do).
>
> Presumably these are sdh and shi, but it is very strange that we cannot find
> the superblock on either of them.
> When was the last time the machine was rebooted prio to the date given -9th
> Jun?
> Do you have boot logs from that time? What lines contain 'md'??
> Particularly "bind" lines will show you exactly which devices were included.
>
> Maybe also try
>
> od -x /dev/sdh | grep '4efc a92b'
>
> If the superblock is at some strange location, that might find it.
>
> NeilBrown
>
May 3 15:07:34 brain kernel: [ 1.778123] md: bind<sda>
May 3 15:07:34 brain kernel: [ 2.434201] md: bind<sdb>
May 3 15:07:34 brain kernel: [ 2.459872] md: bind<sdc>
May 3 15:07:34 brain kernel: [ 3.178179] md: bind<sdd>
May 3 15:07:34 brain kernel: [ 3.808668] md: bind<sdf>
May 3 15:07:34 brain kernel: [ 3.818330] md: bind<sdg>
May 3 15:07:34 brain kernel: [ 3.993520] md: bind<sdl>
May 3 15:07:34 brain kernel: [ 4.496568] md: bind<sdi>
May 3 15:07:34 brain kernel: [ 4.499792] md: bind<sdj>
May 3 15:07:34 brain kernel: [ 4.503025] md: bind<sdh>
May 3 15:07:34 brain kernel: [ 4.749289] md: raid6 personality registered for level 6
May 3 15:07:34 brain kernel: [ 4.750109] md: raid5 personality registered for level 5
May 3 15:07:34 brain kernel: [ 4.750894] md: raid4 personality registered for level 4
May 3 15:07:34 brain kernel: [ 4.752556] md/raid:md0: device sdh operational as raid disk 8
May 3 15:07:34 brain kernel: [ 4.753329] md/raid:md0: device sdj operational as raid disk 6
May 3 15:07:34 brain kernel: [ 4.754071] md/raid:md0: device sdi operational as raid disk 7
May 3 15:07:34 brain kernel: [ 4.754803] md/raid:md0: device sdg operational as raid disk 5
May 3 15:07:34 brain kernel: [ 4.755525] md/raid:md0: device sdf operational as raid disk 4
May 3 15:07:34 brain kernel: [ 4.756247] md/raid:md0: device sdd operational as raid disk 3
May 3 15:07:34 brain kernel: [ 4.756965] md/raid:md0: device sdc operational as raid disk 2
May 3 15:07:34 brain kernel: [ 4.757675] md/raid:md0: device sdb operational as raid disk 1
May 3 15:07:34 brain kernel: [ 4.758368] md/raid:md0: device sda operational as raid disk 0
May 3 15:07:34 brain kernel: [ 4.759359] md/raid:md0: allocated 9616kB
May 3 15:07:34 brain kernel: [ 4.760128] md/raid:md0: raid level 6 active with 9 out of 9 devices, algorithm 2
May 3 15:07:34 brain kernel: [ 4.760944] created bitmap (22 pages) for device md0
May 3 15:07:34 brain kernel: [ 4.762166] md0: bitmap initialized from disk: read 2 pages, set 0 of 44711 bits
May 3 15:07:34 brain kernel: [ 4.775618] md0: detected capacity change from 0 to 21003210260480
May 3 15:07:34 brain kernel: [ 4.785806] md0: unknown partition table
May 3 15:07:34 brain kernel: [ 4.787191] md: bind<sdk>
May 3 15:07:34 brain kernel: [ 5.563969] md: linear personality registered for level -1
May 3 15:07:34 brain kernel: [ 5.564882] md: multipath personality registered for level -4
May 3 15:07:34 brain kernel: [ 5.565700] md: raid0 personality registered for level 0
May 3 15:07:34 brain kernel: [ 5.566577] md: raid1 personality registered for level 1
May 3 15:07:34 brain kernel: [ 5.569815] md: raid10 personality registered for level 10
May 3 15:07:42 brain kernel: [ 15.024766] type=1400 audit(1399122462.862:10): apparmor="STATUS" operation="profile_load" name="/usr/sbin/clamd" pid=2129 comm="apparmor_parser"
May 3 16:30:25 brain kernel: [ 1.148254] md: bind<sda>
May 3 16:30:25 brain kernel: [ 1.171116] md: bind<sdb>
May 3 16:30:25 brain kernel: [ 1.190905] md: bind<sdc>
May 3 16:30:25 brain kernel: [ 1.210995] md: bind<sdd>
May 3 16:30:25 brain kernel: [ 1.218889] md: bind<sdf>
May 3 16:30:25 brain kernel: [ 1.235742] md: bind<sde>
May 3 16:30:25 brain kernel: [ 1.796822] md: bind<sdk>
May 3 16:30:25 brain kernel: [ 2.300024] md: bind<sdh>
May 3 16:30:25 brain kernel: [ 2.307113] md: bind<sdj>
May 3 16:30:25 brain kernel: [ 2.310379] md: bind<sdg>
May 3 16:30:25 brain kernel: [ 2.313719] md: bind<sdi>
May 3 16:30:25 brain kernel: [ 2.559622] md: raid6 personality registered for level 6
May 3 16:30:25 brain kernel: [ 2.560418] md: raid5 personality registered for level 5
May 3 16:30:25 brain kernel: [ 2.561176] md: raid4 personality registered for level 4
May 3 16:30:25 brain kernel: [ 2.562814] md/raid:md0: device sdi operational as raid disk 6
May 3 16:30:25 brain kernel: [ 2.563557] md/raid:md0: device sdg operational as raid disk 8
May 3 16:30:25 brain kernel: [ 2.564277] md/raid:md0: device sdh operational as raid disk 7
May 3 16:30:25 brain kernel: [ 2.564985] md/raid:md0: device sde operational as raid disk 4
May 3 16:30:25 brain kernel: [ 2.565686] md/raid:md0: device sdf operational as raid disk 5
May 3 16:30:25 brain kernel: [ 2.566378] md/raid:md0: device sdd operational as raid disk 3
May 3 16:30:25 brain kernel: [ 2.567067] md/raid:md0: device sdc operational as raid disk 2
May 3 16:30:25 brain kernel: [ 2.567749] md/raid:md0: device sdb operational as raid disk 1
May 3 16:30:25 brain kernel: [ 2.568409] md/raid:md0: device sda operational as raid disk 0
May 3 16:30:25 brain kernel: [ 2.569355] md/raid:md0: allocated 9616kB
May 3 16:30:25 brain kernel: [ 2.570077] md/raid:md0: raid level 6 active with 9 out of 9 devices, algorithm 2
May 3 16:30:25 brain kernel: [ 2.570869] created bitmap (22 pages) for device md0
May 3 16:30:25 brain kernel: [ 2.572057] md0: bitmap initialized from disk: read 2 pages, set 0 of 44711 bits
May 3 16:30:25 brain kernel: [ 2.595329] md0: detected capacity change from 0 to 21003210260480
May 3 16:30:25 brain kernel: [ 2.601758] md0: unknown partition table
May 3 16:30:25 brain kernel: [ 3.183200] md: linear personality registered for level -1
May 3 16:30:25 brain kernel: [ 3.184010] md: multipath personality registered for level -4
May 3 16:30:25 brain kernel: [ 3.184764] md: raid0 personality registered for level 0
May 3 16:30:25 brain kernel: [ 3.185666] md: raid1 personality registered for level 1
May 3 16:30:25 brain kernel: [ 3.188831] md: raid10 personality registered for level 10
May 3 15:07:34 brain kernel: [ 1.778123] md: bind<sda>
May 3 15:07:34 brain kernel: [ 2.434201] md: bind<sdb>
May 3 15:07:34 brain kernel: [ 2.459872] md: bind<sdc>
May 3 15:07:34 brain kernel: [ 3.178179] md: bind<sdd>
May 3 15:07:34 brain kernel: [ 3.808668] md: bind<sdf>
May 3 15:07:34 brain kernel: [ 3.818330] md: bind<sdg>
May 3 15:07:34 brain kernel: [ 3.993520] md: bind<sdl>
May 3 15:07:34 brain kernel: [ 4.496568] md: bind<sdi>
May 3 15:07:34 brain kernel: [ 4.499792] md: bind<sdj>
May 3 15:07:34 brain kernel: [ 4.503025] md: bind<sdh>
May 3 15:07:34 brain kernel: [ 4.749289] md: raid6 personality registered for level 6
May 3 15:07:34 brain kernel: [ 4.750109] md: raid5 personality registered for level 5
May 3 15:07:34 brain kernel: [ 4.750894] md: raid4 personality registered for level 4
May 3 15:07:34 brain kernel: [ 4.752556] md/raid:md0: device sdh operational as raid disk 8
May 3 15:07:34 brain kernel: [ 4.753329] md/raid:md0: device sdj operational as raid disk 6
May 3 15:07:34 brain kernel: [ 4.754071] md/raid:md0: device sdi operational as raid disk 7
May 3 15:07:34 brain kernel: [ 4.754803] md/raid:md0: device sdg operational as raid disk 5
May 3 15:07:34 brain kernel: [ 4.755525] md/raid:md0: device sdf operational as raid disk 4
May 3 15:07:34 brain kernel: [ 4.756247] md/raid:md0: device sdd operational as raid disk 3
May 3 15:07:34 brain kernel: [ 4.756965] md/raid:md0: device sdc operational as raid disk 2
May 3 15:07:34 brain kernel: [ 4.757675] md/raid:md0: device sdb operational as raid disk 1
May 3 15:07:34 brain kernel: [ 4.758368] md/raid:md0: device sda operational as raid disk 0
May 3 15:07:34 brain kernel: [ 4.759359] md/raid:md0: allocated 9616kB
May 3 15:07:34 brain kernel: [ 4.760128] md/raid:md0: raid level 6 active with 9 out of 9 devices, algorithm 2
May 3 15:07:34 brain kernel: [ 4.760944] created bitmap (22 pages) for device md0
May 3 15:07:34 brain kernel: [ 4.762166] md0: bitmap initialized from disk: read 2 pages, set 0 of 44711 bits
May 3 15:07:34 brain kernel: [ 4.775618] md0: detected capacity change from 0 to 21003210260480
May 3 15:07:34 brain kernel: [ 4.785806] md0: unknown partition table
May 3 15:07:34 brain kernel: [ 4.787191] md: bind<sdk>
May 3 15:07:34 brain kernel: [ 5.563969] md: linear personality registered for level -1
May 3 15:07:34 brain kernel: [ 5.564882] md: multipath personality registered for level -4
May 3 15:07:34 brain kernel: [ 5.565700] md: raid0 personality registered for level 0
May 3 15:07:34 brain kernel: [ 5.566577] md: raid1 personality registered for level 1
May 3 15:07:34 brain kernel: [ 5.569815] md: raid10 personality registered for level 10
May 3 15:07:42 brain kernel: [ 15.024766] type=1400 audit(1399122462.862:10): apparmor="STATUS" operation="profile_load" name="/usr/sbin/clamd" pid=2129 comm="apparmor_parser"
May 3 16:30:25 brain kernel: [ 1.148254] md: bind<sda>
May 3 16:30:25 brain kernel: [ 1.171116] md: bind<sdb>
May 3 16:30:25 brain kernel: [ 1.190905] md: bind<sdc>
May 3 16:30:25 brain kernel: [ 1.210995] md: bind<sdd>
May 3 16:30:25 brain kernel: [ 1.218889] md: bind<sdf>
May 3 16:30:25 brain kernel: [ 1.235742] md: bind<sde>
May 3 16:30:25 brain kernel: [ 1.796822] md: bind<sdk>
May 3 16:30:25 brain kernel: [ 2.300024] md: bind<sdh>
May 3 16:30:25 brain kernel: [ 2.307113] md: bind<sdj>
May 3 16:30:25 brain kernel: [ 2.310379] md: bind<sdg>
May 3 16:30:25 brain kernel: [ 2.313719] md: bind<sdi>
May 3 16:30:25 brain kernel: [ 2.559622] md: raid6 personality registered for level 6
May 3 16:30:25 brain kernel: [ 2.560418] md: raid5 personality registered for level 5
May 3 16:30:25 brain kernel: [ 2.561176] md: raid4 personality registered for level 4
May 3 16:30:25 brain kernel: [ 2.562814] md/raid:md0: device sdi operational as raid disk 6
May 3 16:30:25 brain kernel: [ 2.563557] md/raid:md0: device sdg operational as raid disk 8
May 3 16:30:25 brain kernel: [ 2.564277] md/raid:md0: device sdh operational as raid disk 7
May 3 16:30:25 brain kernel: [ 2.564985] md/raid:md0: device sde operational as raid disk 4
May 3 16:30:25 brain kernel: [ 2.565686] md/raid:md0: device sdf operational as raid disk 5
May 3 16:30:25 brain kernel: [ 2.566378] md/raid:md0: device sdd operational as raid disk 3
May 3 16:30:25 brain kernel: [ 2.567067] md/raid:md0: device sdc operational as raid disk 2
May 3 16:30:25 brain kernel: [ 2.567749] md/raid:md0: device sdb operational as raid disk 1
May 3 16:30:25 brain kernel: [ 2.568409] md/raid:md0: device sda operational as raid disk 0
May 3 16:30:25 brain kernel: [ 2.569355] md/raid:md0: allocated 9616kB
May 3 16:30:25 brain kernel: [ 2.570077] md/raid:md0: raid level 6 active with 9 out of 9 devices, algorithm 2
May 3 16:30:25 brain kernel: [ 2.570869] created bitmap (22 pages) for device md0
May 3 16:30:25 brain kernel: [ 2.572057] md0: bitmap initialized from disk: read 2 pages, set 0 of 44711 bits
May 3 16:30:25 brain kernel: [ 2.595329] md0: detected capacity change from 0 to 21003210260480
May 3 16:30:25 brain kernel: [ 2.601758] md0: unknown partition table
May 3 16:30:25 brain kernel: [ 3.183200] md: linear personality registered for level -1
May 3 16:30:25 brain kernel: [ 3.184010] md: multipath personality registered for level -4
May 3 16:30:25 brain kernel: [ 3.184764] md: raid0 personality registered for level 0
May 3 16:30:25 brain kernel: [ 3.185666] md: raid1 personality registered for level 1
May 3 16:30:25 brain kernel: [ 3.188831] md: raid10 personality registered for level 10
May 3 16:30:32 brain kernel: [ 13.703548] type=1400 audit(1399127432.542:9): apparmor="STATUS" operation="profile_load" name="/usr/sbin/clamd" pid=2212 comm="apparmor_parser"
