linux.news@xxxxxxxxxxx wrote, On 20.04.2013 00:56:
Maarten wrote, On 18.04.2013 15:58:
On 18/04/13 15:17, Ben Bucksch wrote:
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware
failure.
* The second array resynced fine, further proving that the
disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy,
1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count
(only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data,
without
resync.
I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the
result
is):
# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq]
mdadm: failed to RUN_ARRAY /dev/md0:
mdadm: Not enough devices to start the array.
At this point, does dmesg show anything pointing to that input/output
error ? The procedure is correct
[dmesg]
The problem is:
md: kicking non-fresh sdl from array!
thus:
raid5: not enough operational devices for md0 (2/8 failed)
So, the question is: How do I convince md not to be so anal retentive
and prevent me from accessing any of my data? The drive ***is fine***,
has practically all the data (I don't care about these 2 events), just
use it already. Nobody seems to know the magic shell commands to do that.
Good news:
In my desperation, I now ran the following dangerous command:
mdadm --create /dev/md0 --assume-clean --level=raid5 -n 8 --chunk=64
--layout=left-symmetric --metadata=0.90 /dev/sdj missing /dev/sdl
/dev/sd[mopnq]
and that worked. I can read my files again, without problem, all is happy.
Before doing that, I saved the superblock, using (no warranty!):
1. mdadm -E /dev/sdj
2. "Used Dev Size" (in KB) * 1024 / 64 - 1 (use this as <skip blocks>)
3. dd if=/dev/sdl of=/root/sdj.mdsuperblock ibs=64 skip=<skip blocks>
---
Thanks, Maarten and Oliver, for your help and moral support.
---
I still maintain that all of this represents 2 design bugs in the md
implementation:
1. ejecting devices out that are working
1.1. individual sectors not readable/writable, but rest of device working
(This is very common these days with large drives)
1.2. temporary errors, e.g. disk not connected, loose cable, bad
controller etc.
1.3. Linux distro upgrade, no disk problem at all (my case)
2. not allowing me to re-add ejected disks, with data, without resync
The result of this is:
1. a device is ejected for no good reasons
2. a resync is triggered
3. the resync discovers a disk that is *really* broken
I am left with 2 disks marked "failed", but only 1 actually failed, so
normally I should be able to recover, yet I cannot read anything. This
fails the very definition of RAID5, therefore is a bug. I have to do
risky operations like re-create that can easily destroy all data.
Effectively, md achieves the opposite that is intended: It actively
risks and destroys my data.
I am BEGGING you md raid devs to fix these.
Ben Bucksch
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
