Re: Reshape Failure

[Yu Kuai <yukuai1@xxxxxxxxxxxxxxx>]

Hi,

在 2023/09/04 5:39, Jason Moss 写道:
> Hello,
>
> I recently attempted to add a new drive to my 8-drive RAID 6 array,
> growing it to 9 drives. I've done similar before with the same array,
> having previously grown it from 6 drives to 7 and then from 7 to 8
> with no issues. Drives are WD Reds, most older than 2019, some
> (including the newest) newer, but all confirmed CMR and not SMR.
>
> Process used to expand the array:
> mdadm --add /dev/md0 /dev/sdb1
> mdadm --grow --raid-devices=9 --backup-file=/root/grow_md0.bak /dev/md0
>
> The reshape started off fine, the process was underway, and the volume
> was still usable as expected. However, 15-30 minutes into the reshape,
> I lost access to the contents of the drive. Checking /proc/mdstat, the
> reshape was stopped at 0.6% with the counter not incrementing at all.
> Any process accessing the array would just hang until killed. I waited

What kernel version are you using? And it'll be very helpful if you can
collect the stack of all stuck thread. There is a known deadlock for
raid5 related to reshape, and it's fixed in v6.5:

https://lore.kernel.org/r/20230512015610.821290-6-yukuai1@xxxxxxxxxxxxxxx

> a half hour and there was still no further change to the counter. At
> this point, I restarted the server and found that when it came back up
> it would begin reshaping again, but only very briefly, under 30
> seconds, but the counter would be increasing during that time.
>
> I searched furiously for ideas and tried stopping and reassembling the
> array, assembling with an invalid-backup flag, echoing "frozen" then
> "reshape" to the sync_action file, and echoing "max" to the sync_max
> file. Nothing ever seemed to make a difference.

Don't do this before v6.5, echo "reshape" while reshape is still in
progress will corrupt your data:

https://lore.kernel.org/r/20230512015610.821290-3-yukuai1@xxxxxxxxxxxxxxx

Thanks,
Kuai

> Here is where I slightly panicked, worried that I'd borked my array,
> and powered off the server again and disconnected the new drive that
> was just added, assuming that since it was the change, it may be the
> problem despite having burn-in tested it, and figuring that I'll rush
> order a new drive, so long as the reshape continues and I can just
> rebuild onto a new drive once the reshape finishes. However, this made
> no difference and the array continued to not rebuild.
>
> Much searching later, I'd found nothing substantially different then
> I'd already tried and one of the common threads in other people's
> issues was bad drives, so I ran a self-test against each of the
> existing drives and found one drive that failed the read test.
> Thinking I had the culprit now, I dropped that drive out of the array
> and assembled the array again, but the same behavior persists. The
> array reshapes very briefly, then completely stops.
>
> Down to 0 drives of redundancy (in the reshaped section at least), not
> finding any new ideas on any of the forums, mailing list, wiki, etc,
> and very frustrated, I took a break, bought all new drives to build a
> new array in another server and restored from a backup. However, there
> is still some data not captured by the most recent backup that I would
> like to recover, and I'd also like to solve the problem purely to
> understand what happened and how to recover in the future.
>
> Is there anything else I should try to recover this array, or is this
> a lost cause?
>
> Details requested by the wiki to follow and I'm happy to collect any
> further data that would assist. /dev/sdb is the new drive that was
> added, then disconnected. /dev/sdh is the drive that failed a
> self-test and was removed from the array.
>
> Thank you in advance for any help provided!
>
>
> $ uname -a
> Linux Blyth 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC
> 2023 x86_64 x86_64 x86_64 GNU/Linux
>
> $ mdadm --version
> mdadm - v4.2 - 2021-12-30
>
>
> $ sudo smartctl -H -i -l scterc /dev/sda
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68EUZN0
> Serial Number:    WD-WCC4N7AT7R7X
> LU WWN Device Id: 5 0014ee 268545f93
> Firmware Version: 82.00A82
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:27:55 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sda
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68EUZN0
> Serial Number:    WD-WCC4N7AT7R7X
> LU WWN Device Id: 5 0014ee 268545f93
> Firmware Version: 82.00A82
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:16 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sdb
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68EUZN0
> Serial Number:    WD-WXG1A8UGLS42
> LU WWN Device Id: 5 0014ee 2b75ef53b
> Firmware Version: 80.00A80
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:19 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sdc
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68EUZN0
> Serial Number:    WD-WCC4N4HYL32Y
> LU WWN Device Id: 5 0014ee 2630752f8
> Firmware Version: 82.00A82
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:20 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sdd
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68N32N0
> Serial Number:    WD-WCC7K1FF6DYK
> LU WWN Device Id: 5 0014ee 2ba952a30
> Firmware Version: 82.00A82
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Form Factor:      3.5 inches
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-3 T13/2161-D revision 5
> SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:21 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sde
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68EUZN0
> Serial Number:    WD-WCC4N5ZHTRJF
> LU WWN Device Id: 5 0014ee 2b88b83bb
> Firmware Version: 82.00A82
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:22 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sdf
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68AX9N0
> Serial Number:    WD-WMC1T3804790
> LU WWN Device Id: 5 0014ee 6036b6826
> Firmware Version: 80.00A80
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:23 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sdg
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68EUZN0
> Serial Number:    WD-WMC4N0H692Z9
> LU WWN Device Id: 5 0014ee 65af39740
> Firmware Version: 82.00A82
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:24 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sdh
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68EUZN0
> Serial Number:    WD-WMC4N0K5S750
> LU WWN Device Id: 5 0014ee 6b048d9ca
> Firmware Version: 82.00A82
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Rotation Rate:    5400 rpm
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:24 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> $ sudo smartctl -H -i -l scterc /dev/sdi
> smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-76-generic] (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family:     Western Digital Red
> Device Model:     WDC WD30EFRX-68AX9N0
> Serial Number:    WD-WMC1T1502475
> LU WWN Device Id: 5 0014ee 058d2e5cb
> Firmware Version: 80.00A80
> User Capacity:    3,000,592,982,016 bytes [3.00 TB]
> Sector Sizes:     512 bytes logical, 4096 bytes physical
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   ACS-2 (minor revision not indicated)
> SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is:    Sun Sep  3 13:28:27 2023 PDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
>
> $ sudo mdadm --examine /dev/sda
> /dev/sda:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sda1
> /dev/sda1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0xd
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856376832 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247728 sectors, after=14336 sectors
>            State : clean
>      Device UUID : 8ca60ad5:60d19333:11b24820:91453532
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124311040 (118.55 GiB 127.29 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 23:12:08 2023
>    Bad Block Log : 512 entries available at offset 24 sectors - bad
> blocks present.
>         Checksum : b6d8f4d1 - correct
>           Events : 181105
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 7
>     Array State : AA.AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sdb
> /dev/sdb:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sdb1
> /dev/sdb1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x5
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856376832 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247728 sectors, after=14336 sectors
>            State : clean
>      Device UUID : 386d3001:16447e43:4d2a5459:85618d11
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124207104 (118.45 GiB 127.19 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 00:02:59 2023
>    Bad Block Log : 512 entries available at offset 24 sectors
>         Checksum : b544a39 - correct
>           Events : 181077
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 8
>     Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sdc
> /dev/sdc:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sdc1
> /dev/sdc1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0xd
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856376832 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247720 sectors, after=14336 sectors
>            State : clean
>      Device UUID : 1798ec4f:72c56905:4e74ea61:2468db75
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124311040 (118.55 GiB 127.29 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 23:12:08 2023
>    Bad Block Log : 512 entries available at offset 72 sectors - bad
> blocks present.
>         Checksum : 88d8b8fc - correct
>           Events : 181105
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 4
>     Array State : AA.AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sdd
> /dev/sdd:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sdd1
> /dev/sdd1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x5
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856376832 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247728 sectors, after=14336 sectors
>            State : clean
>      Device UUID : a198095b:f54d26a9:deb3be8f:d6de9be1
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124311040 (118.55 GiB 127.29 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 23:12:08 2023
>    Bad Block Log : 512 entries available at offset 24 sectors
>         Checksum : d1471d9d - correct
>           Events : 181105
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 6
>     Array State : AA.AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sde
> /dev/sde:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sde1
> /dev/sde1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x5
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856376832 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247720 sectors, after=14336 sectors
>            State : clean
>      Device UUID : acf7ba2e:35d2fa91:6b12b0ce:33a73af5
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124311040 (118.55 GiB 127.29 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 23:12:08 2023
>    Bad Block Log : 512 entries available at offset 72 sectors
>         Checksum : e05d0278 - correct
>           Events : 181105
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 5
>     Array State : AA.AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sdf
> /dev/sdf:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sdf1
> /dev/sdf1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x5
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856373760 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247720 sectors, after=14336 sectors
>            State : clean
>      Device UUID : 31e7b86d:c274ff45:aa6dab50:2ff058c6
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124311040 (118.55 GiB 127.29 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 23:12:08 2023
>    Bad Block Log : 512 entries available at offset 72 sectors
>         Checksum : 26792cc0 - correct
>           Events : 181105
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 0
>     Array State : AA.AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sdg
> /dev/sdg:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sdg1
> /dev/sdg1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x5
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856373760 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247720 sectors, after=14336 sectors
>            State : clean
>      Device UUID : 74476ce7:4edc23f6:08120711:ba281425
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124311040 (118.55 GiB 127.29 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 23:12:08 2023
>    Bad Block Log : 512 entries available at offset 72 sectors
>         Checksum : 6f67d179 - correct
>           Events : 181105
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 1
>     Array State : AA.AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sdh
> /dev/sdh:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sdh1
> /dev/sdh1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0xd
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856373760 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247720 sectors, after=14336 sectors
>            State : clean
>      Device UUID : 31c08263:b135f0f5:763bc86b:f81d7296
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124207104 (118.45 GiB 127.19 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 20:09:14 2023
>    Bad Block Log : 512 entries available at offset 72 sectors - bad
> blocks present.
>         Checksum : b7696b68 - correct
>           Events : 181089
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 2
>     Array State : AAAAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --examine /dev/sdi
> /dev/sdi:
>     MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> $ sudo mdadm --examine /dev/sdi1
> /dev/sdi1:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x5
>       Array UUID : 440dc11e:079308b1:131eda79:9a74c670
>             Name : Blyth:0  (local to host Blyth)
>    Creation Time : Tue Aug  4 23:47:57 2015
>       Raid Level : raid6
>     Raid Devices : 9
>
>   Avail Dev Size : 5856373760 sectors (2.73 TiB 3.00 TB)
>       Array Size : 20497268736 KiB (19.09 TiB 20.99 TB)
>    Used Dev Size : 5856362496 sectors (2.73 TiB 3.00 TB)
>      Data Offset : 247808 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=247720 sectors, after=14336 sectors
>            State : clean
>      Device UUID : ac1063fc:d9d66e6d:f3de33da:b396f483
>
> Internal Bitmap : 8 sectors from superblock
>    Reshape pos'n : 124311040 (118.55 GiB 127.29 GB)
>    Delta Devices : 1 (8->9)
>
>      Update Time : Tue Jul 11 23:12:08 2023
>    Bad Block Log : 512 entries available at offset 72 sectors
>         Checksum : 23b6d024 - correct
>           Events : 181105
>
>           Layout : left-symmetric
>       Chunk Size : 512K
>
>     Device Role : Active device 3
>     Array State : AA.AAAAA. ('A' == active, '.' == missing, 'R' == replacing)
>
> $ sudo mdadm --detail /dev/md0
> /dev/md0:
>             Version : 1.2
>          Raid Level : raid6
>       Total Devices : 9
>         Persistence : Superblock is persistent
>
>               State : inactive
>     Working Devices : 9
>
>       Delta Devices : 1, (-1->0)
>           New Level : raid6
>          New Layout : left-symmetric
>       New Chunksize : 512K
>
>                Name : Blyth:0  (local to host Blyth)
>                UUID : 440dc11e:079308b1:131eda79:9a74c670
>              Events : 181105
>
>      Number   Major   Minor   RaidDevice
>
>         -       8        1        -        /dev/sda1
>         -       8      129        -        /dev/sdi1
>         -       8      113        -        /dev/sdh1
>         -       8       97        -        /dev/sdg1
>         -       8       81        -        /dev/sdf1
>         -       8       65        -        /dev/sde1
>         -       8       49        -        /dev/sdd1
>         -       8       33        -        /dev/sdc1
>         -       8       17        -        /dev/sdb1
>
> $ cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md0 : inactive sdb1[9](S) sdi1[4](S) sdf1[0](S) sdg1[1](S) sdh1[3](S)
> sda1[8](S) sdd1[7](S) sdc1[6](S) sde1[5](S)
>        26353689600 blocks super 1.2
>
> unused devices: <none>
>
> .