OK, after reseating drives and removing the three definitely bad ones, I think the hardware is stable again now. So now I have a problem with the five-drive array I had set up in the mean time. All five drives are there, but one is a bit behind the others in its event count and last update time. Here's the mdadm --examine output: /dev/sdb: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 149c0025:e7c5da3a:62b7a318:4ca57af7 Name : storage1.2 Creation Time : Wed Jul 11 14:50:06 2012 Raid Level : raid6 Raid Devices : 5 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB) Array Size : 17581590528 (8383.56 GiB 9001.77 GB) Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : active Device UUID : 56e9ce91:c5df8850:2105c86d:c9c710a1 Internal Bitmap : 8 sectors from superblock Update Time : Wed Jul 11 15:19:31 2012 Checksum : 80c0762 - correct Events : 276 Layout : left-symmetric Chunk Size : 1024K Device Role : Active device 0 Array State : AAAAA ('A' == active, '.' == missing) /dev/sdj: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 149c0025:e7c5da3a:62b7a318:4ca57af7 Name : storage1.2 Creation Time : Wed Jul 11 14:50:06 2012 Raid Level : raid6 Raid Devices : 5 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB) Array Size : 17581590528 (8383.56 GiB 9001.77 GB) Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : active Device UUID : db72c8d7:672760b4:572dc944:fc7c151b Internal Bitmap : 8 sectors from superblock Update Time : Wed Jul 11 15:29:52 2012 Checksum : 11ec5fef - correct Events : 357 Layout : left-symmetric Chunk Size : 1024K Device Role : Active device 1 Array State : .AAAA ('A' == active, '.' == missing) /dev/sdk: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 149c0025:e7c5da3a:62b7a318:4ca57af7 Name : storage1.2 Creation Time : Wed Jul 11 14:50:06 2012 Raid Level : raid6 Raid Devices : 5 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB) Array Size : 17581590528 (8383.56 GiB 9001.77 GB) Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : active Device UUID : b12fefdd:74914e6e:9f3ca2bd:8b433e34 Internal Bitmap : 8 sectors from superblock Update Time : Wed Jul 11 15:29:52 2012 Checksum : 64035caa - correct Events : 357 Layout : left-symmetric Chunk Size : 1024K Device Role : Active device 2 Array State : .AAAA ('A' == active, '.' == missing) /dev/sdl: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 149c0025:e7c5da3a:62b7a318:4ca57af7 Name : storage1.2 Creation Time : Wed Jul 11 14:50:06 2012 Raid Level : raid6 Raid Devices : 5 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB) Array Size : 17581590528 (8383.56 GiB 9001.77 GB) Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : active Device UUID : db387f8a:383c26f4:4012a3ec:12c7679e Internal Bitmap : 8 sectors from superblock Update Time : Wed Jul 11 15:29:52 2012 Checksum : 2f9569c2 - correct Events : 357 Layout : left-symmetric Chunk Size : 1024K Device Role : Active device 3 Array State : .AAAA ('A' == active, '.' == missing) /dev/sdm: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 149c0025:e7c5da3a:62b7a318:4ca57af7 Name : storage1.2 Creation Time : Wed Jul 11 14:50:06 2012 Raid Level : raid6 Raid Devices : 5 Avail Dev Size : 5860531120 (2794.52 GiB 3000.59 GB) Array Size : 17581590528 (8383.56 GiB 9001.77 GB) Used Dev Size : 5860530176 (2794.52 GiB 3000.59 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : active Device UUID : ac50fe77:91ce387a:e819a38d:4d56a734 Internal Bitmap : 8 sectors from superblock Update Time : Wed Jul 11 15:29:52 2012 Checksum : da66aace - correct Events : 357 Layout : left-symmetric Chunk Size : 1024K Device Role : Active device 4 Array State : .AAAA ('A' == active, '.' == missing) Now, a simple assemble fails: root@dev-storage1:~# mdadm --assemble /dev/md/storage1.2 /dev/sd{b,j,k,l,m} mdadm: /dev/md/storage1.2 assembled from 4 drives - not enough to start the array while not clean - consider --force. root@dev-storage1:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md127 : inactive sdj[1](S) sdm[4](S) sdl[3](S) sdk[2](S) sdb[0](S) 14651327800 blocks super 1.2 unused devices: <none> (Well, md127 exists, but I don't know how to "start" it). So let's try using --force as it suggests: root@dev-storage1:~# mdadm -S /dev/md127 mdadm: stopped /dev/md127 root@dev-storage1:~# mdadm --assemble --force /dev/md/storage1.2 /dev/sd{b,j,k,l,m} mdadm: /dev/md/storage1.2 has been started with 4 drives (out of 5). root@dev-storage1:~# cat /proc/mdstatPersonalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md127 : active raid6 sdj[1] sdm[4] sdl[3] sdk[2] 8790795264 blocks super 1.2 level 6, 1024k chunk, algorithm 2 [5/4] [_UUUU] bitmap: 22/22 pages [88KB], 65536KB chunk unused devices: <none> root@dev-storage1:~# Now I have a 4-drive degraded RAID6, /dev/sdb isn't even listed (even though I gave it on the command line). Is this correct? Is the next thing to do to add the 5th drive into it manually? root@dev-storage1:~# mdadm --manage --re-add /dev/md127 /dev/sdb mdadm: re-added /dev/sdb root@dev-storage1:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md127 : active raid6 sdb[0] sdj[1] sdm[4] sdl[3] sdk[2] 8790795264 blocks super 1.2 level 6, 1024k chunk, algorithm 2 [5/4] [_UUUU] [>....................] recovery = 1.1% (32854540/2930265088) finish=952.5min speed=50692K/sec bitmap: 22/22 pages [88KB], 65536KB chunk unused devices: <none> That seems to have worked, can someone just confirm that's the right sequence of things to do though. This is a test system, next time I do this might be for real :-) Cheers, Brian. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html