Hello,
the kernel version is:
root@ubuntu:~# uname -a
Linux ubuntu 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux
mdadm version is:
root@ubuntu:~# mdadm -V
mdadm - v3.1.4 - 31st August 2010
Examining the three array components:
root@ubuntu:~# mdadm -E /dev/sd{a,b,c}
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b5802763:fd4790dd:ee8bdeb2:2418097f
Name : vc:zvp_1123
Creation Time : Mon Jun 6 21:10:38 2011
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 41940992 (20.00 GiB 21.47 GB)
Array Size : 83879936 (40.00 GiB 42.95 GB)
Used Dev Size : 41939968 (20.00 GiB 21.47 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 8db90071:be80216e:09468262:1f5046b1
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Jun 6 21:10:46 2011
Checksum : 2e424556 - correct
Events : 10
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : A.A ('A' == active, '.' == missing)
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b5802763:fd4790dd:ee8bdeb2:2418097f
Name : vc:zvp_1123
Creation Time : Mon Jun 6 21:10:38 2011
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 41940992 (20.00 GiB 21.47 GB)
Array Size : 83879936 (40.00 GiB 42.95 GB)
Used Dev Size : 41939968 (20.00 GiB 21.47 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9f41313b:b1aa70f8:6cf0ca2f:c6ea0a64
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Jun 6 21:10:44 2011
Checksum : 2d23c61 - correct
Events : 8
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA ('A' == active, '.' == missing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x3
Array UUID : b5802763:fd4790dd:ee8bdeb2:2418097f
Name : vc:zvp_1123
Creation Time : Mon Jun 6 21:10:38 2011
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 41940992 (20.00 GiB 21.47 GB)
Array Size : 83879936 (40.00 GiB 42.95 GB)
Used Dev Size : 41939968 (20.00 GiB 21.47 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Recovery Offset : 999424 sectors
State : active
Device UUID : 61189a9d:ec082cea:a3ba32fb:800fe84b
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Jun 6 21:10:46 2011
Checksum : a47a059 - correct
Events : 10
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : A.A ('A' == active, '.' == missing)
Details about the array:
root@ubuntu:~# mdadm -Q --detail /dev/md1123
/dev/md1123:
Version : 1.2
Creation Time : Mon Jun 6 21:10:38 2011
Raid Level : raid5
Array Size : 41939968 (40.00 GiB 42.95 GB)
Used Dev Size : 20969984 (20.00 GiB 21.47 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Jun 6 21:10:46 2011
State : active, FAILED
Active Devices : 1
Working Devices : 2
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Name : vc:zvp_1123
UUID : b5802763:fd4790dd:ee8bdeb2:2418097f
Events : 10
Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 16 1 faulty spare rebuilding /dev/sdb
3 8 32 2 spare rebuilding /dev/sdc
Basically, the thing is that the faulty (and the rebuilding spare)
component are not kicked out of the array, and the array is stuck in
this state.
Thanks,
Alex.
2011/6/6 Nagilum <nagilum@xxxxxxxxxxx>:
> Make sure you provide all relevant details such as kernel version, mdadm
> version and maybe also mdadm -E /dev/sd{a,b,c}, mdadm -Q --detail /dev/md0,
> ..
>
> ----- Message from alex.bolshoy@xxxxxxxxx ---------
> Date: Sun, 5 Jun 2011 22:41:55 +0300
> From: Alexander Lyakas <alex.bolshoy@xxxxxxxxx>
> Subject: RAID5: failing an active component during spare rebuild - arrays
> hangs
> To: linux-raid@xxxxxxxxxxxxxxx
>
>
>> Hello everybody,
>> I am testing a scenario, in which I create a RAID5 with three devices:
>> /dev/sd{a,b,c}. Since I don't supply --force to mdadm during creation,
>> it treats the array as degraded and starts rebuilding the sdc as a
>> spare. This is as documented.
>>
>> Then I do --fail on /dev/sda. I understand that at this point my data
>> is gone, but I think should still be able to tear down the array.
>>
>> Sometimes I see that /dev/sda is kicked from the array as faulty, and
>> /dev/sdc is also removed and marked as a spare. Then I am able to tear
>> down the array.
>>
>> But sometimes, it looks like the system hits some kind of a deadlock.
>> mdadm --detail produces:
>>
>> Update Time : Sun Jun 5 21:54:34 2011
>> State : active, FAILED
>> Active Devices : 1
>> Working Devices : 2
>> Failed Devices : 1
>> Spare Devices : 1
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Name : ubuntu:zvp_1123
>> UUID : 48a15fb6:b6410bb9:a2ca173e:0092032c
>> Events : 67
>>
>> Number Major Minor RaidDevice State
>> 0 8 0 0 faulty spare rebuilding /dev/sda
>> 1 8 16 1 active sync /dev/sdb
>> 3 8 32 2 spare rebuilding /dev/sdc
>>
>> So the faulty device and the spare are not kicked out of the array. At
>> this point I am unable to do anything with the array:
>>
>> root@ubuntu:~# sudo mdadm --stop /dev/md1123
>> mdadm: failed to stop array /dev/md1123: Device or resource busy
>> Perhaps a running process, mounted filesystem or active volume group?
>> root@ubuntu:~# sudo mdadm /dev/md1123 --remove /dev/sda
>> mdadm: hot remove failed for /dev/sda: Device or resource busy
>> root@ubuntu:~# sudo mdadm /dev/md1123 --remove /dev/sdb
>> mdadm: hot remove failed for /dev/sdb: Device or resource busy
>> root@ubuntu:~# sudo mdadm /dev/md1123 --remove /dev/sdc
>> mdadm: hot remove failed for /dev/sdc: Device or resource busy
>>
>> This is happening on ubuntu-natty, with mdadm - v3.1.4 - 31st August 2010.
>> Looking at some code in mdadm/Detail.c, it looks like /dev/sda has
>> been marked only as MD_DISK_FAULTY, but has not yet been kicked out of
>> the array. The "spare" and "rebuilding" prints also result from that.
>>
>> Same thing also happens (sometimes) when I manually initiate resync
>> (by writing 'repair' to 'sync_action'), and later manually failing one
>> of the devices. Then I also saw messages like this in the syslog:
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350454] INFO: task
>> md1123_resync:7993 blocked for more than 120 seconds.
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350552] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350644] md1123_resync D
>> 0000000000000000 0 7993 2 0x00000004
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350647] ffff8800b56b1cd0
>> 0000000000000046 ffff8800b56b1fd8 ffff8800b56b0000
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350649] 0000000000013d00
>> ffff880036c09a98 ffff8800b56b1fd8 0000000000013d00
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350652] ffff8800b7f1adc0
>> ffff880036c096e0 ffff8800b56b1cb0 ffff880036c56610
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350654] Call Trace:
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350657] [<ffffffff81492885>]
>> md_do_sync+0xb45/0xc90
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350660] [<ffffffff81087940>] ?
>> autoremove_wake_function+0x0/0x40
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350663] [<ffffffff8107861b>] ?
>> recalc_sigpending+0x1b/0x50
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350665] [<ffffffff8148c516>]
>> md_thread+0x116/0x150
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350667] [<ffffffff8148c400>] ?
>> md_thread+0x0/0x150
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350669] [<ffffffff810871f6>]
>> kthread+0x96/0xa0
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350672] [<ffffffff8100cde4>]
>> kernel_thread_helper+0x4/0x10
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350674] [<ffffffff81087160>] ?
>> kthread+0x0/0xa0
>> Jun 5 21:42:00 ubuntu kernel: [ 2280.350676] [<ffffffff8100cde0>] ?
>> kernel_thread_helper+0x0/0x10
>>
>> This is pretty easy for me to reproduce.
>>
>> Basically, I would like to know what the user is expected to do when
>> more than one RAID5 array component fails during rebuild/resync.
>>
>> Thanks,
>> Alex.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
> ----- End message from alex.bolshoy@xxxxxxxxx -----
>
>
>
> ========================================================================
> # _ __ _ __ http://www.nagilum.org/ \n icq://69646724 #
> # / |/ /__ ____ _(_) /_ ____ _ nagilum@xxxxxxxxxxx \n +491776461165 #
> # / / _ `/ _ `/ / / // / ' \ Amiga (68k/PPC): AOS/NetBSD/Linux #
> # /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/ Mac (PPC): MacOS-X / NetBSD /Linux #
> # /___/ x86: FreeBSD/Linux/Solaris/Win2k ARM9: EPOC EV6 #
> ========================================================================
>
>
> ----------------------------------------------------------------
> cakebox.homeunix.net - all the machine one needs..
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
