Thanks, info below.
Linux debian 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1
(2019-09-20) x86_64 GNU/Linux
mdadm - v4.1 - 2018-10-01
/dev/sda
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PBJBNNET
LU WWN Device Id: 5 000cca 23de17ca4
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:45:04 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdb
=== START OF INFORMATION SECTION ===
Device Model: MB4000GCWDC
Serial Number: Z1Z8YWB0
LU WWN Device Id: 5 000c50 07ba725b1
Firmware Version: HPGH
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:46:12 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdc
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCHP96XB
LU WWN Device Id: 5 000cca 24cd7c5eb
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:46:39 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdd
=== START OF INFORMATION SECTION ===
Device Model: WL4000GSA6472E
Serial Number: WOL240331724
LU WWN Device Id: 5 0014ee 0588c0835
Firmware Version: A.01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:46:58 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sde
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCGSLX3B
LU WWN Device Id: 5 000cca 24ccabb82
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:47:15 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdf
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PAGLKHEW
LU WWN Device Id: 5 000cca 22bc8705d
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:47:31 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdg
=== START OF INFORMATION SECTION ===
Device Model: MB4000GCWDC
Serial Number: Z1Z8YWJN
LU WWN Device Id: 5 000c50 07ba717cf
Firmware Version: HPGH
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:47:46 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdh
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCGK79AB
LU WWN Device Id: 5 000cca 24cc7d5b7
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:48:01 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdi
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MG03ACAxxx(Y) Enterprise HDD
Device Model: TOSHIBA MG03ACA400
Serial Number: 5443K65YF
LU WWN Device Id: 5 000039 58b8009aa
Firmware Version: FL1A
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:15 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdj
=== START OF INFORMATION SECTION ===
Device Model: WL4000GSA6472E
Serial Number: WOL240332040
LU WWN Device Id: 5 0014ee 25ddc3495
Firmware Version: 01.01A01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:28 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdk
=== START OF INFORMATION SECTION ===
Device Model: MB4000GCWDC
Serial Number: Z1Z8XQ9J
LU WWN Device Id: 5 000c50 07b9764b6
Firmware Version: HPGH
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:41 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdl
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PAJWABKT
LU WWN Device Id: 5 000cca 22be89bf7
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:54 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdm
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCG1JAVB
LU WWN Device Id: 5 000cca 24cc0b1bf
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:06 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdn
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PAJK4VMS
LU WWN Device Id: 5 000cca 22be3fb8e
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:22 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdo
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCG62VNB
LU WWN Device Id: 5 000cca 24cc2c4f6
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:34 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdp
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCGSNTTB
LU WWN Device Id: 5 000cca 24ccac29d
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:53 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdq
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PAGTSZES
LU WWN Device Id: 5 000cca 22bcb42ee
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:50:05 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdr
=== START OF INFORMATION SECTION ===
Device Model: WL4000GSA6472E
Serial Number: WOL240331985
LU WWN Device Id: 5 0014ee 003311232
Firmware Version: 01.01A01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:50:18 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sda:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 220e5a63:3311984e:f28d4b05:593c684e
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : 6c62a047 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : c9b754df:84118329:c4e1a36b:bd4c7f83
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 5eb7ae75 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 10
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 0b7f323c:5bcbd1c3:89ef2cd2:fa501410
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 49024193 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 17
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 8c0905ef:8e190062:0a5135ab:5ca3cfa0
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 3c6ce1f - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sde:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 896fcc0c:f8b4b6ab:d8056a44:558d0093
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : f6aa6e5a - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 14
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdf:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : a9bbd48d:24117e4b:f0b0c881:4765bc46
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 89499a5 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdg:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 01be0b84:bc5ab713:b1352502:de23ddfa
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : fb8228f1 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdh:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 32f81397:6f0115a4:c3fa2d02:c078170e
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : b22b23ce - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 16
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdi:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 0afa52a2:0f7f4f1f:14e924da:d06d9015
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 181486a1 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdj:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : db1d31c6:cde52a67:9e4c48cc:18bc5dcb
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 2bbec2fc - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdk:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 3d9fab1f:172871cc:02215dee:f6c525be
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : ff5c655d - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdl:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 114ed6e6:9ab8d767:1fcfeacb:4c408814
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : 95de0ba8 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdm:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : c8daba50:67137919:9bead1ab:88d3d498
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 159762fe - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 15
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdn:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : b0893bc1:40c9b3ad:fdf4f722:3acef4c6
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : bf98ccc6 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdo:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : c07db065:7316a61b:aac57004:2641fc5f
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 4c8051b0 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 13
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdp:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdp1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x47
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
Recovery Offset : 4561114112 sectors
State : active
Device UUID : 5843bab9:17b0f8ae:f7712bee:41a23f6c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 39b7d25a - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdq:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 45b14d19:6d4b9afc:e51c3747:c46230b8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 7c0c3300 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdr:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdr1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : f27bbac9:00333c9b:c68170ea:a3507ccd
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : 83a03807 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
lsdrv
**Warning** The following utility(ies) failed to execute:
sginfo
lsusb
Some information may be missing.
PCI [mpt3sas] 05:00.0 Serial Attached SCSI controller: LSI Logic /
Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
└scsi 0:x:x:x [Empty]
PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10
Family) 4 port SATA IDE Controller #1
└scsi 1:x:x:x [Empty]
PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10
Family) 2 port SATA IDE Controller #2
└scsi 3:x:x:x [Empty]
PCI [pata_marvell] 09:00.0 IDE interface: Marvell Technology Group
Ltd. 88SE6121 SATA II / PATA Controller (rev b2)
└scsi 5:x:x:x [Empty]
USB [usb-storage] {13618102022400B0}
└scsi 8:0:0:0 ADATA USB Flash Drive
└sds 7.24g [65:32] Partitioned (dos)
├sds1 1.24g [65:33] vfat 'D-LIVE_10_2' {3E9E-7588}
│├Mounted as /dev/sds1 @ /run/live/persistence/sds1
│└Mounted as /dev/sds1 @ /usr/lib/live/mount/persistence/sds1
└sds2 6.00g [65:34] ext3 'persistence' {bfde904d-48e4-3741-969a-5ad1f0033243}
├Mounted as /dev/sds2 @ /run/live/persistence/sds2
└Mounted as /dev/sds2 @ /usr/lib/live/mount/persistence/sds2
PCI [mpt3sas] 07:00.0 Serial Attached SCSI controller: LSI Logic /
Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
├phy-7:0:12 scsi 7:0:0:0 ATA HITACHI HUS72404 {PBJBNNET}
│└sda 3.64t [8:0] Partitioned (gpt)
│ └sda1 3.64t [8:1] MD raid6 (3/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ │ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
│ └VG super 40.02t 0 free {EqBsuH-weZl-urAI-aO2f-PiO6-rAO3-J8m1F9}
│ ├dm-1 39.73t [253:1] LV nova ext4 {d1359e47-522e-41f5-bb38-8d8d40826e9e}
│ └dm-0 300.00g [253:0] LV star ext4 {dde10d6e-5e4d-4196-8abe-d36d863c97cd}
├phy-7:0:14 scsi 7:0:1:0 ATA MB4000GCWDC {Z1Z8YWB0}
│└sdb 3.64t [8:16] Partitioned (gpt)
│ └sdb1 3.64t [8:17] MD raid6 (10/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:15 scsi 7:0:2:0 ATA Hitachi HUS72404 {PCHP96XB}
│└sdc 3.64t [8:32] Partitioned (gpt)
│ └sdc1 3.64t [8:33] MD raid6 (17/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:18 scsi 7:0:3:0 ATA WL4000GSA6472E {WOL240331724}
│└sdd 3.64t [8:48] Partitioned (gpt)
│ └sdd1 3.64t [8:49] MD raid6 (1/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:19 scsi 7:0:4:0 ATA HITACHI HUS72404 {PAGLKHEW}
│└sdf 3.64t [8:80] Partitioned (gpt)
│ └sdf1 3.64t [8:81] MD raid6 (4/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:20 scsi 7:0:5:0 ATA MB4000GCWDC {Z1Z8YWJN}
│└sdg 3.64t [8:96] Partitioned (gpt)
│ └sdg1 3.64t [8:97] MD raid6 (9/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:21 scsi 7:0:6:0 ATA Hitachi HUS72404 {PCGK79AB}
│└sdh 3.64t [8:112] Partitioned (gpt)
│ └sdh1 3.64t [8:113] MD raid6 (16/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:23 scsi 7:0:7:0 ATA TOSHIBA MG03ACA4 {5443K65YF}
│└sdi 3.64t [8:128] Partitioned (gpt)
│ └sdi1 3.64t [8:129] MD raid6 (7/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:25 scsi 7:0:8:0 ATA WL4000GSA6472E {WOL240332040}
│└sdj 3.64t [8:144] Partitioned (gpt)
│ └sdj1 3.64t [8:145] MD raid6 (0/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:26 scsi 7:0:9:0 ATA Hitachi HUS72404 {PAJWABKT}
│└sdl 3.64t [8:176] Partitioned (gpt)
│ └sdl1 3.64t [8:177] MD raid6 (5/18 (13)) (w/
sdo1,sdn1,sdm1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:27 scsi 7:0:10:0 ATA Hitachi HUS72404 {PCG1JAVB}
│└sdm 3.64t [8:192] Partitioned (gpt)
│ └sdm1 3.64t [8:193] MD raid6 (15/18 (13)) (w/
sdo1,sdn1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:28 scsi 7:0:11:0 ATA Hitachi HUS72404 {PCGSLX3B}
│└sde 3.64t [8:64] Partitioned (gpt)
│ └sde1 3.64t [8:65] MD raid6 (14/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:29 scsi 7:0:12:0 ATA MB4000GCWDC {Z1Z8XQ9J}
│└sdk 3.64t [8:160] Partitioned (gpt)
│ └sdk1 3.64t [8:161] MD raid6 (11/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:30 scsi 7:0:13:0 ATA HITACHI HUS72404 {PAJK4VMS}
│└sdn 3.64t [8:208] Partitioned (gpt)
│ └sdn1 3.64t [8:209] MD raid6 (2/18 (13)) (w/
sdo1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:31 scsi 7:0:14:0 ATA Hitachi HUS72404 {PCG62VNB}
│└sdo 3.64t [8:224] Partitioned (gpt)
│ └sdo1 3.64t [8:225] MD raid6 (13/18 (13)) (w/
sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:32 scsi 7:0:15:0 ATA Hitachi HUS72404 {PCGSNTTB}
│└sdp 3.64t [8:240] Partitioned (gpt)
│ └sdp1 3.64t [8:241] MD raid6 (12/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1)
spare 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:33 scsi 7:0:16:0 ATA HITACHI HUS72404 {PAGTSZES}
│└sdq 3.64t [65:0] Partitioned (gpt)
│ └sdq1 3.64t [65:1] MD raid6 (8/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
└phy-7:0:34 scsi 7:0:17:0 ATA WL4000GSA6472E {WOL240331985}
└sdr 3.64t [65:16] Partitioned (gpt)
└sdr1 3.64t [65:17] MD raid6 (6/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
└md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
Other Block Devices
├loop0 610.27m [7:0] squashfs
│├Mounted as /dev/loop0 @ /run/live/rootfs/filesystem.squashfs
│└Mounted as /dev/loop0 @ /usr/lib/live/mount/rootfs/filesystem.squashfs
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
└loop7 0.00k [7:7] Empty/Unknown
mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid6 sde1[16] sda1[3] sdj1[0] sdr1[6]
sdl1[5] sdi1[7] sdf1[4] sdo1[17] sdh1[14] sdq1[8] sdc1[13] sdn1[2]
sdm1[15] sdd1[1] sdp1[18] sdg1[9] sdk1[12] sdb1[10]
42975736320 blocks super 1.2 level 6, 512k chunk, algorithm 18
[18/17] [UUUUUUUUUUUU_UUUUU]
resync=PENDING
bitmap: 17/30 pages [68KB], 65536KB chunk
unused devices: <none>
Half the drives actually had SCT/ECR disabled. I ran the script which
set all the drives to 7s and tried reshaping again but still stalling.
On Sat, Mar 6, 2021 at 1:20 PM antlists <antlists@xxxxxxxxxxxxxxx> wrote:
>
> https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
>
> You've obviously covered some of this - glitches with Ubuntu and old
> mdadm are common but this clearly isn't that ...
>
> Please give us the information that page asks for - it probably won't
> help much, but some things sometimes jump out ...
>
> Cheers,
> Wol
>
> On 06/03/2021 11:36, apfc123@xxxxxxxxx wrote:
> > Hello,
> >
> > Reshape is currently stalled at 58%. If I reboot the system, the array
> > is started in auto-read-only mode with resync pending. I can freeze
> > the sync and mount the filesystem read only to access the data.
> >
> > When it first stalled and rebooted, I ran extended smart tests on all
> > 18 drives and one came back with read error. I ddrescued it to a new
> > one (99.99% rescued) and swapped it out, but the reshape still stalls
> > immediately when setting the array to --readwrite.
> >
> > The raid5 array started out with 12 drives. I added 6 more drives then
> > grew it to 18 drives and raid level 6 at the same time. This migration
> > was started with mdadm 4.1 running on kernel 4.19. I've tried booting
> > with debian testing running kernel 5.10 (didn't check mdadm version),
> > and also archlinux running 5.11 but with the same results.
> >
> > /dev/md0:
> > Version : 1.2
> > Creation Time : Tue Dec 17 01:51:38 2019
> > Raid Level : raid6
> > Array Size : 42975736320 (40984.86 GiB 44007.15 GB)
> > Used Dev Size : 3906885120 (3725.90 GiB 4000.65 GB)
> > Raid Devices : 18
> > Total Devices : 18
> > Persistence : Superblock is persistent
> >
> > Intent Bitmap : Internal
> >
> > Update Time : Sat Mar 6 09:41:58 2021
> > State : clean, degraded, resyncing (PENDING)
> > Active Devices : 17
> > Working Devices : 18
> > Failed Devices : 0
> > Spare Devices : 1
> >
> > Layout : left-symmetric-6
> > Chunk Size : 512K
> >
> > Consistency Policy : bitmap
> >
> > Delta Devices : 5, (13->18)
> > New Layout : left-symmetric
> >
> > Name : debian:0 (local to host debian)
> > UUID : 2a0d5568:ea53b429:30df79c9:e7559668
> > Events : 303246
> >
> > Number Major Minor RaidDevice State
> > 0 8 129 0 active sync /dev/sdi1
> > 1 8 49 1 active sync /dev/sdd1
> > 2 8 209 2 active sync /dev/sdn1
> > 3 8 1 3 active sync /dev/sda1
> > 4 8 65 4 active sync /dev/sde1
> > 5 8 145 5 active sync /dev/sdj1
> > 6 65 17 6 active sync /dev/sdr1
> > 7 8 113 7 active sync /dev/sdh1
> > 8 65 1 8 active sync /dev/sdq1
> > 9 8 81 9 active sync /dev/sdf1
> > 10 8 17 10 active sync /dev/sdb1
> > 12 8 193 11 active sync /dev/sdm1
> > 18 8 241 12 spare rebuilding /dev/sdp1
> > 17 8 225 13 active sync /dev/sdo1
> > 16 8 177 14 active sync /dev/sdl1
> > 15 8 161 15 active sync /dev/sdk1
> > 14 8 97 16 active sync /dev/sdg1
> > 13 8 33 17 active sync /dev/sdc1
> >
> > One of the entries from dmesg the first time reshaping stalled:
> >
> > [105003.994653] INFO: task md0_reshape:3296 blocked for more than 120 seconds.
> > [105003.994916] Tainted: G I 4.19.0-6-amd64 #1
> > Debian 4.19.67-2+deb10u1
> > [105003.995169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > [105003.995434] md0_reshape D 0 3296 2 0x80000000
> > [105003.995436] Call Trace:
> > [105003.995441] ? __schedule+0x2a2/0x870
> > [105003.995442] schedule+0x28/0x80
> > [105003.995448] reshape_request+0x862/0x940 [raid456]
> > [105003.995451] ? finish_wait+0x80/0x80
> > [105003.995454] raid5_sync_request+0x34a/0x3b0 [raid456]
> > [105003.995460] md_do_sync.cold.86+0x3f4/0x911 [md_mod]
> > [105003.995461] ? finish_wait+0x80/0x80
> > [105003.995464] ? __switch_to_asm+0x35/0x70
> > [105003.995467] ? md_rdev_init+0xb0/0xb0 [md_mod]
> > [105003.995471] md_thread+0x94/0x150 [md_mod]
> > [105003.995473] kthread+0x112/0x130
> > [105003.995475] ? kthread_bind+0x30/0x30
> > [105003.995476] ret_from_fork+0x35/0x40
> >
> > I'm not sure where to continue troubleshooting. Been moving drives
> > around in the storage appliance in case there are bad ports on the
> > backplane but doesn't seem to make any difference. I only have one
> > 4-port HBA right now so can't even try another. The storage appliance
> > has 2 controllers and already tried swapping them as well as the SAS
> > cables. Only thing left I can try hardware wise is to install the
> > interposers that go in between the drives and the backplane but not
> > very hopeful.
> >
> > Any help would be greatly appreciated. I don't have the extra space
> > right now to copy everything out and create a new array so really
> > hoping to get past this stalling issue. Thanks.
> >
