Hi. I need some help with recovering from multiple disk failure on a RAID6 array. I had two failed disks and therefore shut down the server and connected new disks. After I powered on the server, another disk got booted out of the array leaving it with only 15 out of 18 working devices, so it won’t start. I ran an offline test with smartctl and the disk that got thrown out of the array seems totally fine. Here is where I think I made a mistake. I use the –re-add command on the disk. Now it is regarded as spare and the array still won’t start. I’ve been reading on https://raid.wiki.kernel.org/index.php/RAID_Recovery and I have tried `–assemble –scan –force –verbose` and manual `–assemble –force` with specifying each drive. Neither of them works (reporting that 15 out of 18 devices is not enough). All drives has the same event count and used dev size, but two of the devices has a lower Avail Dev Size, and a different Data Offset. After a bit of digging in the manual and on different forums I have concluded that the next step for me is to recreate the array using –assume-clean and –data-offset=variable. I have tried a dry run of the command (answering no to “Continue creating array”), and mdadm accepts the parameters without any errors: mdadm --create --assume-clean --level=6 --raid-devices=18 --size=3906763776s --chunk=512K --data-offset=variable /dev/md0 /dev/sdj1:262144s /dev/sdk1:262144s /dev/sdi1:262144s /dev/sdh1:262144s /dev/sdo1:262144s /dev/sdp1:262144s /dev/sdr1:262144s /dev/sdq1:262144s /dev/sdf1:262144s /dev/sdb1:262144ss /dev/sdg1:262144s /dev/sdd1:262144s /dev/sdm1:262144s /dev/sdf2:241664s missing missing /dev/sdc2:241664s /dev/sdc1:262144s mdadm: /dev/sdj1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdk1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdi1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdh1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdo1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdp1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdr1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdq1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdf1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdb1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: partition table exists on /dev/sdb1 but will be lost or meaningless after creating array mdadm: /dev/sdg1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdd1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdm1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdf2 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdc2 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 mdadm: /dev/sdc1 appears to be part of a raid array: level=raid6 devices=18 ctime=Wed Nov 14 22:53:28 2012 Continue creating array? N My only worries now are the size and data-offset parameters. According to the man page, the size should be specified in KiloBytes. It was KibiBytes previously. The Used Device Size of all array members is 3906763776 sectors (1862.89 GiB 2000.26 GB). Should I convert the sectors into KiloBytes or does mdadm support using sectors as unit for –size and data-offset? It is not mentioned in the manual, but I’ve seen it being used on different forum threads and mdadm does not blow up if I try using it. Any other suggestions?
