I started with a 4x4TB disk RAID5 array and, over a few years changed all the drives to 8TB (WD Red - I hadn't seen the warnings before now, but it looks like these ones are OK). I then successfully migrated it to RAID6, but it then had a non-standard layout, so I ran: sudo mdadm --grow /dev/md0 --raid-devices=4 --backup-file=/root/raid5backup --layout=normalize After a few days it reached 99% complete, but then the "hours remaining" counter started counting up. After a few days I had to power the system down before I could get a backup of the non-critical data (Couldn't get hold of enough storage quickly enough, but it wouldn't be catastrophic to lose it), and now the four drives are in standby, with the array thinking it is RAID0. Running: sudo mdadm --assemble /dev/md0 /dev/sd[bcde] responds with: mdadm: /dev/md0 assembled from 4 drives - not enough to start the array while not clean - consider --force. It appears to be similar to https://marc.info/?t=155492912100004&r=1&w=2, but before trying --force I was considering using overlay files as I'm not sure of the risk of damage. The set-up process that is documented in the " Recovering a damaged RAID" Wiki article is excellent, however the latter part of the process isn't clear to me. If successful, are the overlay files written to the disk like a virtual machine snapshot, or is the process stopped, the overlays removed and the process repeated, knowing that it now has a low risk of damage? System details follow. Thanks for any help. ============================================================================ ===== user@host:~$ uname -a Linux conan 5.4.0-74-generic #83-Ubuntu SMP Sat May 8 02:35:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ============================================================================ ===== user@host:~$ mdadm --version mdadm - v4.1 - 2018-10-01 ============================================================================ ===== user@host:~$ sudo smartctl -H -i -l scterc /dev/sda smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Samsung based SSDs Device Model: Samsung SSD 860 EVO M.2 250GB Serial Number: S413NX0K707647T LU WWN Device Id: 5 002538 e40528ae8 Firmware Version: RVT21B6Q User Capacity: 250,059,350,016 bytes [250 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: M.2 Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:10 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: Disabled Write: Disabled ============================================================================ ===== user@host:~$ sudo smartctl -H -i -l scterc /dev/sdb smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD80EFAX-68KNBN0 Serial Number: VGJM3NXK LU WWN Device Id: 5 000cca 0bee4dfda Firmware Version: 81.00A81 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:10 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) ============================================================================ ===== user@host:~$ sudo smartctl -H -i -l scterc /dev/sdc smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WD80EFBX-68AZZN0 Serial Number: VRG5YT4K LU WWN Device Id: 5 000cca 0c2c2b5a4 Firmware Version: 85.00A85 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:11 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) ============================================================================ ===== user@host:~$ sudo smartctl -H -i -l scterc /dev/sdd smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD80EFAX-68KNBN0 Serial Number: VAGV1WLL LU WWN Device Id: 5 000cca 099cbd8be Firmware Version: 81.00A81 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:12 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) ============================================================================ ===== user@host:~$ sudo smartctl -H -i -l scterc /dev/sde smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD80EFAX-68LHPN0 Serial Number: 7SJ5W2KW LU WWN Device Id: 5 000cca 252deda87 Firmware Version: 83.H0A83 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:12 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) ============================================================================ ===== user@host:~$ sudo mdadm --examine /dev/sdb /dev/sdb: Magic : a92b4efc Version : 1.2 Feature Map : 0xd Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29 Name : Universe:0 Creation Time : Thu Jul 13 01:11:22 2017 Raid Level : raid6 Raid Devices : 4 Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB) Array Size : 15627793408 (14903.83 GiB 16002.86 GB) Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB) Data Offset : 259072 sectors Super Offset : 8 sectors Unused Space : before=258992 sectors, after=688 sectors State : active Device UUID : eee9201e:d9769906:b6dccda1:b1f35abe Internal Bitmap : 8 sectors from superblock Reshape pos'n : 39006208 (37.20 GiB 39.94 GB) Delta Devices : -1 (5->4) New Layout : left-symmetric Update Time : Fri Jun 18 08:56:43 2021 Bad Block Log : 512 entries available at offset 24 sectors - bad blocks present. Checksum : de1db60e - correct Events : 184251 Layout : left-symmetric-6 Chunk Size : 512K Device Role : Active device 0 Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing) ============================================================================ ===== user@host:~$ sudo mdadm --examine /dev/sdc /dev/sdc: Magic : a92b4efc Version : 1.2 Feature Map : 0xd Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29 Name : Universe:0 Creation Time : Thu Jul 13 01:11:22 2017 Raid Level : raid6 Raid Devices : 4 Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB) Array Size : 15627793408 (14903.83 GiB 16002.86 GB) Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB) Data Offset : 259072 sectors Super Offset : 8 sectors Unused Space : before=258992 sectors, after=688 sectors State : active Device UUID : 7ed45d83:84db8f79:e3aadf4b:a88212d1 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 39006208 (37.20 GiB 39.94 GB) Delta Devices : -1 (5->4) New Layout : left-symmetric Update Time : Fri Jun 18 08:56:43 2021 Bad Block Log : 512 entries available at offset 24 sectors - bad blocks present. Checksum : 731a6e9f - correct Events : 184251 Layout : left-symmetric-6 Chunk Size : 512K Device Role : Active device 1 Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing) ============================================================================ ===== user@host:~$ sudo mdadm --examine /dev/sdd /dev/sdd: Magic : a92b4efc Version : 1.2 Feature Map : 0xd Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29 Name : Universe:0 Creation Time : Thu Jul 13 01:11:22 2017 Raid Level : raid6 Raid Devices : 4 Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB) Array Size : 15627793408 (14903.83 GiB 16002.86 GB) Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB) Data Offset : 259072 sectors Super Offset : 8 sectors Unused Space : before=258992 sectors, after=688 sectors State : active Device UUID : 015b3ea0:9b3a38d2:a860f58a:34c19985 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 39006208 (37.20 GiB 39.94 GB) Delta Devices : -1 (5->4) New Layout : left-symmetric Update Time : Fri Jun 18 08:56:43 2021 Bad Block Log : 512 entries available at offset 24 sectors - bad blocks present. Checksum : dc4048b8 - correct Events : 184251 Layout : left-symmetric-6 Chunk Size : 512K Device Role : Active device 2 Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing) ============================================================================ ===== user@host:~$ sudo mdadm --examine /dev/sde /dev/sde: Magic : a92b4efc Version : 1.2 Feature Map : 0x5 Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29 Name : Universe:0 Creation Time : Thu Jul 13 01:11:22 2017 Raid Level : raid6 Raid Devices : 4 Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB) Array Size : 15627793408 (14903.83 GiB 16002.86 GB) Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB) Data Offset : 259072 sectors Super Offset : 8 sectors Unused Space : before=258992 sectors, after=688 sectors State : active Device UUID : bf9e316b:5910c7ca:1fd799e3:41a349b3 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 39006208 (37.20 GiB 39.94 GB) Delta Devices : -1 (5->4) New Layout : left-symmetric Update Time : Fri Jun 18 08:56:43 2021 Bad Block Log : 512 entries available at offset 24 sectors Checksum : 2616ba80 - correct Events : 184251 Layout : left-symmetric-6 Chunk Size : 512K Device Role : Active device 3 Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing) ============================================================================ ===== user@host:~$ sudo mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Raid Level : raid0 Total Devices : 4 Persistence : Superblock is persistent State : inactive Working Devices : 4 Delta Devices : -1, (1->0) New Level : raid6 New Layout : left-symmetric New Chunksize : 512K Name : Universe:0 UUID : 3eee8746:8a3bf425:afb9b538:daa61b29 Events : 184251 Number Major Minor RaidDevice - 8 64 - /dev/sde - 8 32 - /dev/sdc - 8 48 - /dev/sdd - 8 16 - /dev/sdb ============================================================================ ===== user@host:~$ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 9.1M 1 loop /snap/canonical-livepatch/98 loop1 7:1 0 9.1M 1 loop /snap/canonical-livepatch/99 loop2 7:2 0 99.4M 1 loop /snap/core/11187 loop3 7:3 0 99.2M 1 loop /snap/core/11167 loop4 7:4 0 55.4M 1 loop /snap/core18/2066 loop5 7:5 0 70.4M 1 loop /snap/lxd/19647 loop7 7:7 0 217.5M 1 loop /snap/nextcloud/28088 loop8 7:8 0 67.6M 1 loop /snap/lxd/20326 loop9 7:9 0 217.5M 1 loop /snap/nextcloud/27920 loop10 7:10 0 55.5M 1 loop /snap/core18/2074 sda 8:0 0 232.9G 0 disk +-sda1 8:1 0 512M 0 part /boot/efi L-sda2 8:2 0 232.4G 0 part / sdb 8:16 0 7.3T 0 disk sdc 8:32 0 7.3T 0 disk sdd 8:48 0 7.3T 0 disk sde 8:64 0 7.3T 0 disk ============================================================================ ===== user@host:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : inactive sdc[7](S) sdb[6](S) sde[4](S) sdd[5](S) 31255588192 blocks super 1.2 ============================================================================ ===== user@host:~$ sudo mdadm --stop /dev/md0 mdadm: stopped /dev/md0 user@host:~$ sudo mdadm --assemble /dev/md0 /dev/sd[bcde] mdadm: /dev/md0 assembled from 4 drives - not enough to start the array while not clean - consider --force.
