Hi all, I am sorry for bad English, it is not my primary language. Does solution, described here - "http://marc.info/?l=linux-raid&m=144659416216285&w=2";, matches for mine issue? Please, read issue description below, I am ready to provide additional info. RAID1 (md1; boot), RAID10 (md2; swap, root) and RAID5 (md3; LVM -> ext4 -> data) are placed on 10 disks, each of them is 4TB WD Red with 3 partitions on it. All disks model is WDC WD40EFRX-68WT0N0. sd?1 partitions belongs to md1, sd?2 belongs to md2, sd?3 belongs to md3. sdj caught IO error and was reinitialized under new minor and in-/dev/ IDs, in fact partitions on it was dropped from arrays. sdk1 and sdk2, previous sdj1 and sdj2 accordingly, was readded to their arrays by me, but md3 stay leave sdj3 as member of array, I can't do anything with it, only receiving errors like that: # mdadm /dev/md3 -r detached mdadm: Cannot find 8:147: No such file or directory so sdk3 can't be readded. So I've managed to grow md3 array with command like following, do not remember why and strictly command: # mdadm --grow --raid-devices=11 -add /dev/md3 /dev/sdk3 array start to reshaping, but caught second error and reshaping process stalled with 1536 bytes transferred. Then was executed near two useless commands, like making sdk3 spare device. Now md3 consists 11 devices, was 10, has 2 faulty devices, actually 1 - old sdj3 and sdj3 under new ID - sdk3, which is slightly corrupted by writing 1536 bytes. Seems, that I should and write data back to source block device to do it consistent and able to reboot, recreate array with one corrupted device and resync it. Which data was transferred at reshaping - raw data from first block device in the array, including mdadm metadata (/dev/sd?3), or data from array's summary block device (/dev/md3)? After this all happened, host was not rebooted, stay online, FS on md3 array unavailable, smartctl reports (-H argument), that all disks are healthy. # uname -sr Linux 3.13.0-52-generic # strace -p `pgrep -o mdadm` -f Process 1250 attached select(5, NULL, NULL, [4], {181, 601546} # cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md2 : active raid10 sdk2[12] sdg2[1] sdh2[10] sdi2[5] sdf2[9] sde2[8] sdd2[6] sdc2[4] sdb2[11] sda2[0] 35653120 blocks super 1.2 512K chunks 2 near-copies [10/10] [UUUUUUUUUU] bitmap: 1/1 pages [4KB], 65536KB chunk md1 : active raid1 sdk1[9] sdg1[6] sdh1[7] sdi1[8] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0] 203712 blocks [10/10] [UUUUUUUUUU] bitmap: 0/1 pages [0KB], 65536KB chunk md3 : active raid5 sdk3[14](S) sdg3[6] sdh3[11] sdj3[13](F) sdi3[10] sde3[4] sdf3[5] sdd3[3] sdc3[2] sdb3[12] sda3[0] 35095924224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [11/10] [UUUUUUUU_U_] [>....................] reshape = 0.0% (1536/3899547136) finish=10215829552.5min speed=0K/sec bitmap: 0/15 pages [0KB], 131072KB chunk unused devices: <none> # blkid /dev/sda3 /dev/sdk3 /dev/sda3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="fa7c541d-a25b-abd8-14ff-e1c0c6c1578c" LABEL="ubuntu:3" TYPE="linux_raid_member" /dev/sdk3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="c0155214-a9c8-e378-d368-b6f6735048a8" LABEL="ubuntu:3" TYPE="linux_raid_member" # mdadm --detail /dev/md3 /dev/md3: Version : 1.2 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Array Size : 35095924224 (33470.08 GiB 35938.23 GB) Used Dev Size : 3899547136 (3718.90 GiB 3993.14 GB) Raid Devices : 11 Total Devices : 11 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Nov 6 16:53:06 2015 State : active, FAILED, reshaping Active Devices : 9 Working Devices : 10 Failed Devices : 1 Spare Devices : 1 Layout : left-symmetric Chunk Size : 512K Reshape Status : 0% complete Delta Devices : 1, (10->11) Name : ubuntu:3 UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Events : 21586 Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 12 8 19 1 active sync /dev/sdb3 2 8 35 2 active sync /dev/sdc3 3 8 51 3 active sync /dev/sdd3 4 8 67 4 active sync /dev/sde3 5 8 83 5 active sync /dev/sdf3 6 8 99 6 active sync /dev/sdg3 11 8 115 7 active sync /dev/sdh3 16 0 0 16 removed 10 8 131 9 active sync /dev/sdi3 20 0 0 20 removed 13 8 147 - faulty 14 8 163 - spare /dev/sdk3 # "mdadm --examine" output /dev/sda3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : fa7c541d:a25babd8:14ffe1c0:c6c1578c Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Checksum : ac275763 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdb3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : 70afdaad:b0979fb1:1b99ef18:dd58cc3a Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 1c2bf759 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 1 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdc3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : 345b0212:6c3a0ae8:d8ed9273:ecc53573 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Checksum : 49c30753 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdd3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : a4b8b425:b0307fe5:9eb8e3ac:ae985311 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Checksum : 3258f890 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 3 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sde3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : 4490843e:d2c5c2e7:d222fa0a:f98b277c Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Checksum : 1656c2d2 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 4 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdf3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : 1d41056f:88149b1a:48e939e5:8c775e23 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Checksum : fb26746a - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 5 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdg3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : 375d85e2:9077dd68:bc1327af:8fee4112 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Checksum : 75b99505 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 6 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdh3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : 67cd2bfa:affa578c:e0c1286a:bd4d2627 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 80c895f3 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 7 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdi3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : a5f2cea8:a89c88a0:0dd3900c:624f0491 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Checksum : 4fda6fb3 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 9 Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdj3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : c0155214:a9c8e378:d368b6f6:735048a8 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 952a55f2 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : spare Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdk3: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e Name : ubuntu:3 Creation Time : Thu Jul 24 16:06:32 2014 Raid Level : raid5 Raid Devices : 11 Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB) Array Size : 38995471360 (37188.98 GiB 39931.36 GB) Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB) Data Offset : 262144 sectors New Offset : 251904 sectors Super Offset : 8 sectors State : clean Device UUID : c0155214:a9c8e378:d368b6f6:735048a8 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 0 Delta Devices : 1 (10->11) Update Time : Fri Nov 6 16:53:06 2015 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 952a55f2 - correct Events : 21586 Layout : left-symmetric Chunk Size : 512K Device Role : spare Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing) # grep'ed smartctl output #1, /dev/sda SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) /dev/sdb SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) /dev/sdc SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) /dev/sdd SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) /dev/sde SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) /dev/sdf SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) /dev/sdg SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) /dev/sdh SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) /dev/sdi SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) /dev/sdk SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) # grep 'sd.3' /proc/partitions 8 3 3899678535 sda3 8 19 3899678535 sdb3 8 35 3899678535 sdc3 8 51 3899678535 sdd3 8 67 3899678535 sde3 8 83 3899678535 sdf3 8 99 3899678535 sdg3 8 115 3899678535 sdh3 8 131 3899678535 sdi3 8 163 3899678535 sdk3 # grep'ed smartctl output #2 /dev/sda 9 Power_On_Hours -O--CK 082 082 000 - 13233 /dev/sdb 9 Power_On_Hours -O--CK 085 085 000 - 11051 /dev/sdc 9 Power_On_Hours -O--CK 085 085 000 - 11379 /dev/sdd 9 Power_On_Hours -O--CK 085 085 000 - 11403 /dev/sde 9 Power_On_Hours -O--CK 081 081 000 - 14233 /dev/sdf 9 Power_On_Hours -O--CK 085 085 000 - 11357 /dev/sdg 9 Power_On_Hours -O--CK 085 085 000 - 11382 /dev/sdh 9 Power_On_Hours -O--CK 085 085 000 - 11097 /dev/sdi 9 Power_On_Hours -O--CK 082 082 000 - 13256 /dev/sdk 9 Power_On_Hours -O--CK 089 089 000 - 8413 # dmesg "http://paste.debian.net/335374/"; -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
