On Mon, 24 Sep 2012 15:37:11 +0200 Jakub Husák <jakub@xxxxxxxxxxx> wrote: > Hi, I have found a serious bug, that affects at least 4-disk md raid10 > with far2 layout. The kernel allows it to run with two failed drives > silently without failing the whole array, despite it's not possible for > it to work correctly because of chunks distribution with far2 layout. > The worst thing about it is that the write IO errors are invisible for > the file system and running processes, the written data are just lost, > only with IO errors reported in dmesg. Even force-reassembling ends up > with clean,degraded array, with TWO disks, ignoring that I've tried to > assemble it with all four devices. Recreating the array with > --assume-clean is the only way to put it together. Why do you say that "the write IO errors are invisible for the filesystem"? They are certainly reported in the kernel logs that you should and I'm sure an application would see them if it checked return status properly. md is behaving as designed here. It deliberately does not fail the whole array, it just fails those blocks which are no longer accessible. NeilBrown > > System: > > Ubuntu 12.04 > Linux version 3.2.0-30-generic (buildd@batsu) (gcc version 4.6.3 > (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #48-Ubuntu SMP Fri Aug 24 16:52:48 UTC > 2012 > mdadm - v3.2.5 - 18th May 2012 > > and > > Debian 6.0 > Linux version 2.6.32-5-xen-amd64 (Debian 2.6.32-35) (dannf@xxxxxxxxxx) > (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Tue Jun 14 12:46:30 UTC 2011 > mdadm - v3.1.4 - 31st August 2010 > > and > > Centos 6.3 > > > How to repeat: > > dd if=/dev/zero of=d0 bs=1M count=100 > dd if=/dev/zero of=d1 bs=1M count=100 > dd if=/dev/zero of=d2 bs=1M count=100 > dd if=/dev/zero of=d3 bs=1M count=100 > losetup -f d0 > losetup -f d1 > losetup -f d2 > losetup -f d3 > > mdadm -C /dev/md0 --level=10 --raid-devices=4 --layout=f2 /dev/loop[0-3] > > dd if=/dev/zero of=/dev/md0 bs=512K count=10 > 10+0 records in > 10+0 records out > 5242880 bytes (5,2 MB) copied, 0,0409824 s, 128 MB/s > > OK > > mdadm /dev/md0 --fail /dev/loop0 > mdadm /dev/md0 --fail /dev/loop3 > > mdadm -D /dev/md0 > /dev/md0: > Version : 1.2 > Creation Time : Mon Sep 24 08:47:10 2012 > Raid Level : raid10 > Array Size : 202752 (198.03 MiB 207.62 MB) > Used Dev Size : 101376 (99.02 MiB 103.81 MB) > Raid Devices : 4 > Total Devices : 4 > Persistence : Superblock is persistent > > Update Time : Mon Sep 24 08:48:55 2012 > State : clean, degraded <<< !!!!!!!! > Active Devices : 2 > Working Devices : 2 > Failed Devices : 2 > Spare Devices : 0 > > Layout : far=2 > Chunk Size : 512K > > Name : koubas-desktop:0 (local to host koubas-desktop) > UUID : 3ea4ded7:c10b1778:dc9f92aa:6e7cb196 > Events : 21 > > Number Major Minor RaidDevice State > 0 0 0 0 removed <<< !!!!!!!! > 1 7 1 1 active sync /dev/loop1 > 2 7 2 2 active sync /dev/loop2 > 3 0 0 3 removed <<< !!!!!!!! > > 0 7 0 - faulty spare /dev/loop0 > 3 7 3 - faulty spare /dev/loop3 > > dd if=/dev/zero of=/dev/md0 bs=512K count=10 > 10+0 records in > 10+0 records out > 5242880 bytes (5,2 MB) copied, 0,0245752 s, 213 MB/s > echo $? > 0 <<< !!!!!!! > > dmesg: > [883011.442366] md/raid10:md0: Disk failure on loop0, disabling device. > [883011.442367] md/raid10:md0: Operation continuing on 3 devices. > [883011.473292] RAID10 conf printout: > [883011.473296] --- wd:3 rd:4 > [883011.473299] disk 0, wo:1, o:0, dev:loop0 > [883011.473301] disk 1, wo:0, o:1, dev:loop1 > [883011.473302] disk 2, wo:0, o:1, dev:loop2 > [883011.473304] disk 3, wo:0, o:1, dev:loop3 > [883011.492046] RAID10 conf printout: > [883011.492051] --- wd:3 rd:4 > [883011.492054] disk 1, wo:0, o:1, dev:loop1 > [883011.492056] disk 2, wo:0, o:1, dev:loop2 > [883011.492058] disk 3, wo:0, o:1, dev:loop3 > [883015.875089] md/raid10:md0: Disk failure on loop3, disabling device. > [883015.875090] md/raid10:md0: Operation continuing on 2 devices. <<< !!!!! > [883015.886686] RAID10 conf printout: > [883015.886692] --- wd:2 rd:4 > [883015.886695] disk 1, wo:0, o:1, dev:loop1 > [883015.886697] disk 2, wo:0, o:1, dev:loop2 > [883015.886699] disk 3, wo:1, o:0, dev:loop3 > [883015.900018] RAID10 conf printout: > [883015.900023] --- wd:2 rd:4 > [883015.900025] disk 1, wo:0, o:1, dev:loop1 > [883015.900027] disk 2, wo:0, o:1, dev:loop2 > ************* "successful" dd follows: ******************* > [883015.903622] quiet_error: 6 callbacks suppressed > [883015.903624] Buffer I/O error on device md0, logical block 50672 > [883015.903628] Buffer I/O error on device md0, logical block 50672 > [883015.903635] Buffer I/O error on device md0, logical block 50686 > [883015.903638] Buffer I/O error on device md0, logical block 50686 > [883015.903669] Buffer I/O error on device md0, logical block 50687 > [883015.903672] Buffer I/O error on device md0, logical block 50687 > [883015.903706] Buffer I/O error on device md0, logical block 50687 > [883015.903710] Buffer I/O error on device md0, logical block 50687 > [883015.903714] Buffer I/O error on device md0, logical block 50687 > [883015.903717] Buffer I/O error on device md0, logical block 50687 > [883052.136435] quiet_error: 6 callbacks suppressed > [883052.136439] Buffer I/O error on device md0, logical block 384 > [883052.136442] lost page write due to I/O error on md0 > [883052.136448] Buffer I/O error on device md0, logical block 385 > [883052.136450] lost page write due to I/O error on md0 > [883052.136454] Buffer I/O error on device md0, logical block 386 > [883052.136456] lost page write due to I/O error on md0 > [883052.136460] Buffer I/O error on device md0, logical block 387 > [883052.136462] lost page write due to I/O error on md0 > [883052.136466] Buffer I/O error on device md0, logical block 388 > [883052.136468] lost page write due to I/O error on md0 > [883052.136472] Buffer I/O error on device md0, logical block 389 > [883052.136474] lost page write due to I/O error on md0 > [883052.136478] Buffer I/O error on device md0, logical block 390 > [883052.136480] lost page write due to I/O error on md0 > [883052.136484] Buffer I/O error on device md0, logical block 391 > [883052.136486] lost page write due to I/O error on md0 > [883052.136492] Buffer I/O error on device md0, logical block 392 > [883052.136494] lost page write due to I/O error on md0 > [883052.136498] Buffer I/O error on device md0, logical block 393 > [883052.136500] lost page write due to I/O error on md0 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html
Attachment:
signature.asc
Description: PGP signature
