Hi!
I think I've got a problem :-/ I have a QNAP NAS with a 8 disks RAID5.
Some days ago I got a "Disk Read/Write Error" on the 8th drive
(/dev/sdh), with the suggestion to replace the disk.
I replaced it, but after a bit the RAID rebuilding failed, and the QNAP
Admin Interface still gives me a "Disk Read/Write Error" on /dev/sdh.
Plus, I can't access to the RAID data anymore :-/
I was following this guide
https://raid.wiki.kernel.org/index.php/RAID_Recovery but, since I
haven't got any backup (I promise I will do them in the future!) I'm
afraid to run any possibly destructive command.
How do you suggest to proceed? I would like to make a RAID excluding the
8th disk in order to mount it and backup important data, but I don't
even know if it is doable :-/ Moreover, looking at `mdadm --examine`
output I see that sdb seems to have problems too, also if QNAP Admin
Interface doesn't report it.
Here some informations about the machine status:
# uname -a
Linux NAS 3.4.6 #1 SMP Thu Sep 12 10:56:51 CST 2013 x86_64 unknown
# mdadm -V
mdadm - v2.6.3 - 20th August 2007
# cat /etc/mdadm.conf
ARRAY /dev/md0
devices=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sdd3,/dev/sde3,/dev/sdf3,/dev/sdg3,/dev/sdh3
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4] [multipath]
md8 : active raid1 sdg2[2](S) sdf2[3](S) sde2[4](S) sdd2[5](S)
sdc2[6](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]
md13 : active raid1 sda4[0] sde4[6] sdf4[5] sdg4[4] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [8/7] [UUUUUUU_]
bitmap: 8/57 pages [32KB], 4KB chunk
md9 : active raid1 sda1[0] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [8/7] [UUUUUUU_]
bitmap: 30/65 pages [120KB], 4KB chunk
unused devices: <none>
# mdadm --examine /dev/sd[abcdefgh]3
/dev/sda3:
Magic : a92b4efc
Version : 00.90.00
UUID : 418e2add:2c4b313b:d12fb7ea:993d5bf7
Creation Time : Fri Jan 20 02:19:47 2012
Raid Level : raid5
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Update Time : Fri Jan 24 17:19:58 2014
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : 982047ab - correct
Events : 0.2944851
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 3 0 active sync /dev/sda3
0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
/dev/sdb3:
Magic : a92b4efc
Version : 00.90.00
UUID : 418e2add:2c4b313b:d12fb7ea:993d5bf7
Creation Time : Fri Jan 20 02:19:47 2012
Raid Level : raid5
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Jan 24 17:09:57 2014
State : active
Active Devices : 7
Working Devices : 8
Failed Devices : 1
Spare Devices : 1
Checksum : 97f3567d - correct
Events : 0.2944837
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 19 1 active sync /dev/sdb3
0 0 8 3 0 active sync /dev/sda3
1 1 8 19 1 active sync /dev/sdb3
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
8 8 8 115 8 spare /dev/sdh3
/dev/sdc3:
Magic : a92b4efc
Version : 00.90.00
UUID : 418e2add:2c4b313b:d12fb7ea:993d5bf7
Creation Time : Fri Jan 20 02:19:47 2012
Raid Level : raid5
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Update Time : Fri Jan 24 17:19:58 2014
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : 982047cf - correct
Events : 0.2944851
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 35 2 active sync /dev/sdc3
0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
/dev/sdd3:
Magic : a92b4efc
Version : 00.90.00
UUID : 418e2add:2c4b313b:d12fb7ea:993d5bf7
Creation Time : Fri Jan 20 02:19:47 2012
Raid Level : raid5
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Update Time : Fri Jan 24 17:19:58 2014
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : 982047e1 - correct
Events : 0.2944851
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 51 3 active sync /dev/sdd3
0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
/dev/sde3:
Failed Devices : 2
Spare Devices : 0
Checksum : 982047f3 - correct
Events : 0.2944851
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 67 4 active sync /dev/sde3
0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
/dev/sdf3:
Magic : a92b4efc
Version : 00.90.00
UUID : 418e2add:2c4b313b:d12fb7ea:993d5bf7
Creation Time : Fri Jan 20 02:19:47 2012
Raid Level : raid5
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Update Time : Fri Jan 24 17:19:58 2014
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : 98204805 - correct
Events : 0.2944851
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 83 5 active sync /dev/sdf3
0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
/dev/sdg3:
Magic : a92b4efc
Version : 00.90.00
UUID : 418e2add:2c4b313b:d12fb7ea:993d5bf7
Creation Time : Fri Jan 20 02:19:47 2012
Raid Level : raid5
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Update Time : Fri Jan 24 17:19:58 2014
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : 98204817 - correct
Events : 0.2944851
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 99 6 active sync /dev/sdg3
0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
/dev/sdh3:
Magic : a92b4efc
Version : 00.90.00
UUID : 418e2add:2c4b313b:d12fb7ea:993d5bf7
Creation Time : Fri Jan 20 02:19:47 2012
Raid Level : raid5
Used Dev Size : 1951945600 (1861.52 GiB 1998.79 GB)
Array Size : 13663619200 (13030.64 GiB 13991.55 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Jan 24 17:18:26 2014
State : clean
Active Devices : 6
Working Devices : 7
Failed Devices : 2
Spare Devices : 1
Checksum : 98204851 - correct
Events : 0.2944847
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 115 8 spare /dev/sdh3
0 0 8 3 0 active sync /dev/sda3
1 1 0 0 1 faulty removed
2 2 8 35 2 active sync /dev/sdc3
3 3 8 51 3 active sync /dev/sdd3
4 4 8 67 4 active sync /dev/sde3
5 5 8 83 5 active sync /dev/sdf3
6 6 8 99 6 active sync /dev/sdg3
7 7 0 0 7 faulty removed
8 8 8 115 8 spare /dev/sdh3
# dmesg **edited (removed unuseful parts)**
, wo:0, o:1, dev:sdb2
[ 975.516724] RAID1 conf printout:
[ 975.516728] --- wd:2 rd:2
[ 975.516732] disk 0, wo:0, o:1, dev:sda2
[ 975.516737] disk 1, wo:0, o:1, dev:sdb2
[ 975.516740] RAID1 conf printout:
[ 975.516744] --- wd:2 rd:2
[ 975.516748] disk 0, wo:0, o:1, dev:sda2
[ 975.516753] disk 1, wo:0, o:1, dev:sdb2
[ 977.495709] md: unbind<sdh2>
[ 977.505048] md: export_rdev(sdh2)
[ 977.535277] md/raid1:md9: Disk failure on sdh1, disabling device.
[ 977.575038] disk 2, wo:0, o:1, dev:sdc1
[ 977.575043] disk 3, wo:0, o:1, dev:sdd1
[ 977.575048] disk 4, wo:0, o:1, dev:sde1
[ 977.575053] disk 5, wo:0, o:1, dev:sdf1
[ 977.575058] disk 6, wo:0, o:1, dev:sdg1
[ 979.547149] md: unbind<sdh1>
[ 979.558031] md: export_rdev(sdh1)
[ 979.592646] md/raid1:md13: Disk failure on sdh4, disabling device.
[ 979.592650] md/raid1:md13: Operation continuing on 7 devices.
[ 979.650862] RAID1 conf printout:
[ 979.650869] --- wd:7 rd:8
[ 979.650875] disk 0, wo:0, o:1, dev:sda4
[ 979.650880] disk 1, wo:0, o:1, dev:sdb4
[ 979.650885] disk 2, wo:0, o:1, dev:sdc4
[ 979.650890] disk 3, wo:0, o:1, dev:sdd4
[ 979.650895] disk 4, wo:0, o:1, dev:sdg4
[ 979.650900] disk 5, wo:0, o:1, dev:sdf4
[ 979.650905] disk 6, wo:0, o:1, dev:sde4
[ 979.650911] disk 7, wo:1, o:0, dev:sdh4
[ 979.656024] RAID1 conf printout:
[ 979.656029] --- wd:7 rd:8
[ 979.656034] disk 0, wo:0, o:1, dev:sda4
[ 979.656039] disk 1, wo:0, o:1, dev:sdb4
[ 979.656044] disk 2, wo:0, o:1, dev:sdc4
[ 979.656049] disk 3, wo:0, o:1, dev:sdd4
[ 979.656054] disk 4, wo:0, o:1, dev:sdg4
[ 979.656059] disk 5, wo:0, o:1, dev:sdf4
[ 979.656063] disk 6, wo:0, o:1, dev:sde4
[ 981.604906] md: unbind<sdh4>
[ 981.616035] md: export_rdev(sdh4)
[ 981.753058] md/raid:md0: Disk failure on sdh3, disabling device.
[ 981.753062] md/raid:md0: Operation continuing on 6 devices.
[ 983.765852] md: unbind<sdh3>
[ 983.777030] md: export_rdev(sdh3)
[ 1060.094825] journal commit I/O error
[ 1060.099196] journal commit I/O error
[ 1060.103525] journal commit I/O error
[ 1060.108698] journal commit I/O error
[ 1060.116311] journal commit I/O error
[ 1060.123634] journal commit I/O error
[ 1060.127225] journal commit I/O error
[ 1060.130930] journal commit I/O error
[ 1060.137651] EXT4-fs (md0): previous I/O error to superblock detected
[ 1060.178323] Buffer I/O error on device md0, logical block 0
[ 1060.181873] lost page write due to I/O error on md0
[ 1060.185634] EXT4-fs error (device md0): ext4_put_super:849: Couldn't
clean up the journal
[ 1062.662723] md0: detected capacity change from 13991546060800 to 0
[ 1062.666308] md: md0 stopped.
[ 1062.669760] md: unbind<sda3>
[ 1062.681031] md: export_rdev(sda3)
[ 1062.684466] md: unbind<sdg3>
[ 1062.695023] md: export_rdev(sdg3)
[ 1062.698342] md: unbind<sdf3>
[ 1062.709021] md: export_rdev(sdf3)
[ 1062.712310] md: unbind<sde3>
[ 1062.723029] md: export_rdev(sde3)
[ 1062.726245] md: unbind<sdd3>
[ 1062.737022] md: export_rdev(sdd3)
[ 1062.740112] md: unbind<sdc3>
[ 1062.751022] md: export_rdev(sdc3)
[ 1062.753934] md: unbind<sdb3>
[ 1062.764021] md: export_rdev(sdb3)
[ 1063.772687] md: md0 stopped.
[ 1064.782381] md: md0 stopped.
[ 1065.792585] md: md0 stopped.
[ 1066.801668] md: md0 stopped.
[ 1067.812573] md: md0 stopped.
[ 1068.821548] md: md0 stopped.
[ 1069.830667] md: md0 stopped.
[ 1070.839554] md: md0 stopped.
[ 1071.848418] md: md0 stopped.
--
Maurizio De Santis
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
