swiss cheesed raid5 raid recovery?

[Grant Parnell - EverythingLinux <grant@xxxxxxxxxxxxxxxxxxxxxx>]

Thanks heaps Neil. There turns out to be quite a lot of combinations to 
try so I thought I'd try programatically all 96 of them. Essentially 
creating an array and seeing if I can extract a directory listing using 
debugfs. I am curious as to whether the order of the devices matters and 
also what are the consequences of getting the layout wrong. All but one 
were originally set to 'left-symmetric', the other was 'left-asymmetric'.

Anyway... the best I got was 24 cases of 'EXT2 directory corrupted'

Attached is the perl script and log output. Other people may find the 
script of use. Hopefully someone has some other ideas for testing the 
filesystem.

On Tue, 18 May 2004, Neil Brown wrote:

> On Tuesday May 18, grant@xxxxxxxxxxxxxxxxxxxxxx wrote:
> > If there's a proper place to ask, please forward this on.
> 
> linux-raid@xxxxxxxxxxxxxxx would be the best place.
> 
> > 
> > I've got a very silly client who has no backup of this data and would like 
> > it recovered and doesen't mind the wait. Description was that one of the 
> > physical disks had failed for some time (OK so far) ... then recently 
> > another failed although he thinks the 2nd failure was just a small glitch 
> > and it could be put back into service (sounds OK but not great).
> > 
> > What I'd like is to be able to try various combinations of the partitions 
> > to assemble into an array in a read-only fassion (at the partition level!) 
> > so as not to write to the disks under any circumstances.
> 
> What I would do is 
>  - use "mdadm --examine" on each device so as to have a permanent
>    record of what the superblocks contained.
>  - use "mdadm --create" with various combinations of devices, always
>    specifying one slot as "missing"
> 
> This will over-write the superblocks, but nothing else (as there is a
> missing device, no resync can happen).
> 
> If you manage to find some correct configuration, hot-add a fourth
> drive and be happy.
> 
> This does change the UUID, but it is much easier than any alternative.
> 
> There is no tool (to my knowledge) for generally editing md
> superblocks. 
> 
> > 
> > By looking at /dev/md1 so far with hdd3, hde3, hdg3 it barely looks like 
> > an ext2/3 filesystem. Even debugfs is giving up.
> 
> That would have been the best bet, and what "mdadm --assemble --force"
> would have provided.  
> 
> I don't suppose the customer remembers what they did for hdc3.....
> 
> 
> NeilBrown
> 

-- 
---<GRiP>--- 
Grant Parnell - senior consultant
EverythingLinux services - the consultant's backup & tech support.
Web: http://www.everythinglinux.com.au/support.php
We're also busybits.com.au and linuxhelp.com.au and elx.com.au.
Phone 02 8752 6622 to book service or discuss your needs.

ELX or its employees participate in the following:-
OSIA (Open Source Industry Australia) - http://www.osia.net.au
AUUG (Australian Unix Users Group) - http://www.auug.org.au
SLUG (Sydney Linux Users Group) - http://www.slug.org.au
LA (Linux Australia) - http://www.linux.org.au

#!/usr/bin/perl

my @devs = qw ( /dev/hdc3 /dev/hdd3 /dev/hde3 /dev/hdg3 );

for(my $missing=1; $missing<=4; $missing++)
{
	for (my $dev1=0; $dev1<=3; $dev1++)
	{
		for (my $dev2=0; $dev2<=3; $dev2++)
		{
			next if ($dev2 == $dev1);
			for (my $dev3=0; $dev3<=3; $dev3++)
			{
				next if ($dev3 == $dev2 || $dev3 == $dev1);
				#print "missing=$missing, dev1=$dev1, dev2=$dev2, dev3=$dev3\n";
				my $dpos=1;
				my $devlist='';
				for (my $pos=1; $pos<=4; $pos++)
				{
					if($pos==$missing) { $devlist.="missing "; }
					elsif($dpos==1) { $devlist.="$devs[$dev1] "; $dpos++; }
					elsif($dpos==2) { $devlist.="$devs[$dev2] "; $dpos++; }
					elsif($dpos==3) { $devlist.="$devs[$dev3] "; $dpos++; }
				}	
				my $status = check_raid($devlist);
				print "$status\n";
			}
		}
	}
}

sub check_raid
{
	my ($devlist) = @_;
	if (! $devlist) { return "failed command"; }
	# stop the array if started
	my $cmd = "mdadm --manage --stop /dev/md1";
	docmd($cmd);
	# create the array
	$cmd = "echo y | mdadm --create /dev/md1 --level=5 --raid-devices=4 --spare-devices=0 $devlist";
	docmd($cmd);
	# make read-only
	$cmd = "mdadm --misc --readonly /dev/md1";
	docmd($cmd);
	# debugfs to check?
	$cmd = "debugfs -R ls";
	my $result = docmd($cmd);
	print "result: $result\n";
	#if ($result =~ /\/dev\/md1/) { return "fail"; } else { return "ok"; }
	if ($result =~ /:/) { return "fail"; } else { return "ok"; }
}

sub docmd
{
	my ($cmd) = @_;
	print "sh: $cmd\n";
	#return '';

	open(CMD,"$cmd 2>&1|");
	my $result='';
	while(<CMD>)
	{
		$result.=$_;
	}
	close(CMD);
	return $result;
}

Attachment: combo.log.gz
Description: GNU Zip compressed data