On Wed, 9 May 2012 11:10:58 +0200 (CEST) piergiorgio.sartor@xxxxxxxx wrote:
> Hi all,
>
> we're hit by a RAID-5 issue, it seems Ubuntu 12.04 is shipping
> some bugged kernel/mdadm combination.
Buggy kernel. My fault. I think they know and an update should follow.
However I suspect that Ubuntu must be doing something else to cause the
problem to trigger so often. The circumstance that makes it happen should be
extremely rare. It is as though the md array is half-stopped just before
shutdown. If it were completely stopped or not stopped at all, this wouldn't
happen.
>
> Following the other thread about a similar issue, I understood
> it is possible to fix the array without losing data.
Correct.
>
> Problems are:
>
> 1) We do not know the HDD order and it is a 5 disks RAID-5
If you have kernel logs from the last successful boot they would contain
a "RAID conf printout" which would give you the order, but maybe that it on
the RAID-5 array?
If it is you will have to try different permutations until you find one that
works.
> 2) 4 of 5 disks have a data offset of 264 sectors, while the
> fourth one, added later, has 1048 sectors.
Ouch.
It would be easiest to just make a degraded array with the 4 devices with the
same data offset, then add the 5th later.
To get the correct data offset you could either use the same mdadm that the
array was originally built with, or you could get the 'r10-reshape'
branch from git://neil.brown.name/mdadm/ and build that.
Then create the array with --data-offset=132K as well as all the other flags.
However that hasn't been tested extensively so it would be best to test it
elsewhere first. Check that it created the array with correct data-offset
and correct size.
> 3) There is a LVM setup on the array, not a plain filesystem.
That does make it a little more complex but not much.
You would need to activate the LVM, then "fsck -n" the filesystems to check if
you have the devices in the right order.
However this could help you identify the first device quickly.
If you
dd if=/dev/sdXX skip=264 count=1
then for the first device in the array it will show you the textual
description of the LVM setup. For the other devices it will probably be
binary or something unrelated.
>
> Any idea on how can we get the array back without losing any
> data?
Do you know what the chunk size was? Probably 64K if it was an old array.
Maybe 512K though.
I would:
1/ look at old logs if possible to find out the device order
2/ try to remember what the chunk size could be. If you have the exact
used-device size (mdadm -E should give that) you can get an upper limit
for the chunk size by finding the larger power-of-2 which divides it.
3/ Try to identify the first device by looking for LVM metadata.
4/ Make a list of the possible arrangements of devices and possible chunk
sizes based on the info you collected.
5/ Check that you can create an array with a data-offset for 264 sectors
using one of the approaches listed above.
6/ write a script which iterated though the possibilities and re-created the
array then tries to turn on LVM and the fsck. Or maybe iterate by hand.
The command to create an array would be something like
mdadm -C /dev/md0 -l5 -n5 --assume-clean --chunk=64 \
--data-offset=132K /dev/sdX missing /dev/sdY /dev/sdZ /dev/sdW
7/ Find out which arrangement produces least fsck errors, and use that.
>
> At the moment, it seems quite difficult to provide dump of
> "mdadm -E" or similar, since the PC does not boot at all.
> In any case, if necessary we could try to take a picture of
> the screen and send it here or directly per email, if appropriate.
You probably need to boot from a DVD-ROM or similar.
Certainly feel free to post the data you collect and the conclusions you draw
and even the script you write if you would like them reviewed and confirmed.
NeilBrown
Attachment:
signature.asc
Description: PGP signature
