I have a small ubuntu server that I was upgrading the hardware to and in the process lost my raid. I changed the CPU, MOBO, RAM. I added a new-to-this-system SSD also to replace the current SSD(in a future step). I forgot that this new-to-this-system SSD had Windows 10 OS on it and I believe it tried to boot while I was working on hooking up my monitor. So I think that it saw my raid drives and tried to fdisk them. I did mdadm directly to drive and not to a partition(big mistake I know now). So I think the drives were seen as corrupted and fdisk corrected the formatting. I lost my super blocks on 4 of 5 drives. These are shucked external 10TB drives and one even shows up with 'my drive' partition label and 2 files that came with those drives. I want to recover my raid and files but don't want to make it worse. I have not mounted the drives as writable. I think the damage should be limited, but I don't know mdadm well. I have been digging for a few days on options and most advice is generic and bad and I feel would make it worse. I don't know the original order the drives were in. 1 drive is fully intact, probably due to a BIOS sata config not enabling all drives when i first booted. The size makes this impractical to dd onto new disks. The drives were 99% full and I was about to add 2 new drives. Now if i can recover this, i will be starting a new array correctly and transfering files to that. To fix, I have been leaning toward making the drives ready only and using an overlay file. Like here: https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file But i dont understand all the commands well enough to work this for my situation. Seems like since I don't know the original drive arrangement that may be adding an additional level of complexity. If I can figure out the read only and overlay, I still don't know exactly the right way to proceed on the mdadm front. Please anyone who has a handle on a situation like this, let me know what I should do. Thanks **Original command history for array: sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdc /dev/sdd /dev/sde cat /proc/mdstat sudo mkfs.ext4 -F /dev/md0 sudo mkdir -p /media/raid sudo mount /dev/md0 /media/raid df -h -x devtmpfs -x tmpfs cat /proc/mdstat sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf sudo update-initramfs -u sudo umount /dev/md0 sudo umount /dev/md0 -f sudo umount /dev/md0 sudo fsck.ext4 -f /dev/md0 sudo fsck.ext4 sudo fsck.ext4 -f /dev/md0 -p sudo fsck.ext4 -f /dev/md0 -p -y sudo fsck.ext4 -f /dev/md0 -y sudo resize2fs /dev/md0 sudo fdisk -l sudo parted -a optimal /dev/sdf sudo -i mdadm --add /dev/md0 /dev/sdf watch cat /proc/mdstat sudo mdadm --grow /dev/md0 --raid-devices=4 sudo thunar watch cat /proc/mdstat sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf cat /proc/mdstat sudo mount /dev/md0 /media/raid sudo mdadm --assemble --scan sudo mount /dev/md0 /media/raid sudo fdisk -l sudo parted -s -a optimal /dev/sdb mklabel gpt parted /dev/sdb sudo parted /dev/sdb sudo mdadm --add /dev/md0 /dev/sdb1 sudo mdadm --add /dev/md0 /dev/sdb cat /proc/mdstat mdadm --grow --raid-devices=4 /dev/md0 sudo mdadm --grow --raid-devices=4 /dev/md0 sudo mdadm --grow --raid-devices=5 /dev/md0 cat /proc/mdstat sudo e2fsck -f /dev/md0 cat /proc/mdstat sudo resize2fs /dev/md0 cat /proc/mdstat sudo e2fsck -f /dev/md0 sudo resize2fs /dev/md0 **Here are some current details: uname -a Linux server 5.11.0-40-generic #44-Ubuntu SMP Wed Oct 20 16:16:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux mdadm --version mdadm - v4.1 - 2018-10-01 ** sudo smartctl -H -i -l scterc /dev/sda smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.11.0-41-generic] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Ultrastar He10/12 Device Model: WDC WD100EZAZ-11TDBA0 Serial Number: 1EK7U77Z LU WWN Device Id: 5 000cca 27eedd3d5 Firmware Version: 83.H0A83 User Capacity: 10,000,831,348,736 bytes [10.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Nov 30 00:07:28 2021 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: Disabled Write: Disabled ** sudo smartctl -H -i -l scterc /dev/sdb smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.11.0-41-generic] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Ultrastar He10/12 Device Model: WDC WD100EMAZ-00WJTA0 Serial Number: JEHXKMMM LU WWN Device Id: 5 000cca 267db1416 Firmware Version: 83.H0A83 User Capacity: 10,000,831,348,736 bytes [10.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Nov 30 00:08:34 2021 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) ** sudo smartctl -H -i -l scterc /dev/sdc smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.11.0-41-generic] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Ultrastar He10/12 Device Model: WDC WD100EMAZ-00WJTA0 Serial Number: 2YHVAJ8D LU WWN Device Id: 5 000cca 273da10a9 Firmware Version: 83.H0A83 User Capacity: 10,000,831,348,736 bytes [10.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Nov 30 00:11:29 2021 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) ** sudo smartctl -H -i -l scterc /dev/sdd smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.11.0-41-generic] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Ultrastar He10/12 Device Model: WDC WD100EMAZ-00WJTA0 Serial Number: 2YHVABZD LU WWN Device Id: 5 000cca 273da1024 Firmware Version: 83.H0A83 User Capacity: 10,000,831,348,736 bytes [10.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Nov 30 00:11:58 2021 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) ** sudo smartctl -H -i -l scterc /dev/sde smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.11.0-41-generic] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Ultrastar He10/12 Device Model: WDC WD100EMAZ-00WJTA0 Serial Number: 2YHV9GVD LU WWN Device Id: 5 000cca 273da0cbc Firmware Version: 83.H0A83 User Capacity: 10,000,831,348,736 bytes [10.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Nov 30 00:12:53 2021 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) *************** sudo mdadm --examine /dev/sda /dev/sda: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee) sudo mdadm --examine /dev/sda1 /dev/sda1: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 4294967295 (type ff) Partition[1] : 4294967295 sectors at 4294967295 (type ff) Partition[2] : 4294967295 sectors at 4294967295 (type ff) Partition[3] : 740229375 sectors at 4294967295 (type ff) sudo mdadm --examine /dev/sdb /dev/sdb: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee) sudo mdadm --examine /dev/sdb1 mdadm: cannot open /dev/sdb1: No such file or directory sudo mdadm --examine /dev/sdc /dev/sdc: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee) sudo mdadm --examine /dev/sdc1 mdadm: cannot open /dev/sdc1: No such file or directory sudo mdadm --examine /dev/sdd /dev/sdd: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 93e81091:84ba78f0:eb8232d9:c3c995f0 Name : bushserver:0 (local to host bushserver) Creation Time : Fri Nov 16 13:20:25 2018 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 19532616704 (9313.88 GiB 10000.70 GB) Array Size : 39065219072 (37255.50 GiB 40002.78 GB) Used Dev Size : 19532609536 (9313.87 GiB 10000.70 GB) Data Offset : 257024 sectors Super Offset : 8 sectors Unused Space : before=256944 sectors, after=7168 sectors State : clean Device UUID : 2abcf2dc:f786e3fd:d22b7da9:7e8eec53 Internal Bitmap : 8 sectors from superblock Update Time : Sun Nov 28 15:27:11 2021 Bad Block Log : 512 entries available at offset 48 sectors Checksum : e27debbf - correct Events : 213198 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 1 Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing) $ sudo mdadm --examine /dev/sdd1 mdadm: cannot open /dev/sdd1: No such file or directory sudo mdadm --examine /dev/sde /dev/sde: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee) $ sudo mdadm --examine /dev/sde1 mdadm: cannot open /dev/sde1: No such file or directory **************************************************** sudo mdadm --detail /dev/md0 /dev/md0: Version : 1.2 Raid Level : raid0 Total Devices : 1 Persistence : Superblock is persistent State : inactive Working Devices : 1 Name : bushserver:0 (local to host bushserver) UUID : 93e81091:84ba78f0:eb8232d9:c3c995f0 Events : 213198 Number Major Minor RaidDevice - 8 48 - /dev/sdd ******************************************************* ./lsdrv **Warning** The following utility(ies) failed to execute: sginfo pvs lvs Some information may be missing. PCI [nvme] 41:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) └nvme nvme0 Force MP510 {211182930001291838A6} └nvme0n1 447.13g [259:0] Empty/Unknown ├nvme0n1p1 431.03g [259:1] Empty/Unknown │└Mounted as /dev/nvme0n1p1 @ / ├nvme0n1p2 1.00k [259:2] Empty/Unknown └nvme0n1p5 15.87g [259:3] Empty/Unknown PCI [ahci] 00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31) ├scsi 0:0:0:0 ATA WDC WD100EZAZ-11 │└sda 9.10t [8:0] Empty/Unknown │ └sda1 9.10t [8:1] Empty/Unknown ├scsi 1:0:0:0 ATA WDC WD100EMAZ-00 │└sdb 9.10t [8:16] Empty/Unknown ├scsi 3:0:0:0 ATA WDC WD100EMAZ-00 │└sdc 9.10t [8:32] Empty/Unknown ├scsi 4:0:0:0 ATA WDC WD100EMAZ-00 │└sdd 9.10t [8:48] Empty/Unknown │ └md0 0.00k [9:0] MD v1.2 () inactive, None (None) None {00000000:-0000-00:00-0000-:000000000000} │ Empty/Unknown └scsi 5:0:0:0 ATA WDC WD100EMAZ-00 └sde 9.10t [8:64] Empty/Unknown PCI [ahci] 04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) └scsi 6:x:x:x [Empty] USB [usb-storage] Bus 001 Device 004: ID 1d6b:0104 Linux Foundation Multifunction Composite Gadget {CAFEBABE} └scsi 8:0:0:0 Linux File-CD Gadget └sr0 1.00g [11:0] Empty/Unknown Other Block Devices ├loop0 4.00k [7:0] Empty/Unknown │└Mounted as /dev/loop0 @ /snap/bare/5 ├loop1 144.60m [7:1] Empty/Unknown │└Mounted as /dev/loop1 @ /snap/chromium/1810 ├loop2 99.44m [7:2] Empty/Unknown │└Mounted as /dev/loop2 @ /snap/core/11798 ├loop3 99.44m [7:3] Empty/Unknown │└Mounted as /dev/loop3 @ /snap/core/11993 ├loop4 147.80m [7:4] Empty/Unknown │└Mounted as /dev/loop4 @ /snap/chromium/1827 ├loop5 55.49m [7:5] Empty/Unknown │└Mounted as /dev/loop5 @ /snap/core18/2253 ├loop6 55.50m [7:6] Empty/Unknown │└Mounted as /dev/loop6 @ /snap/core18/2246 ├loop7 65.21m [7:7] Empty/Unknown │└Mounted as /dev/loop7 @ /snap/gtk-common-themes/1519 ├loop8 164.76m [7:8] Empty/Unknown │└Mounted as /dev/loop8 @ /snap/gnome-3-28-1804/161 ├loop9 65.10m [7:9] Empty/Unknown │└Mounted as /dev/loop9 @ /snap/gtk-common-themes/1515 ├loop10 162.87m [7:10] Empty/Unknown │└Mounted as /dev/loop10 @ /snap/gnome-3-28-1804/145 ├loop11 0.00k [7:11] Empty/Unknown ├zram0 1.96g [252:0] Empty/Unknown ├zram1 1.96g [252:1] Empty/Unknown ├zram2 1.96g [252:2] Empty/Unknown ├zram3 1.96g [252:3] Empty/Unknown ├zram4 1.96g [252:4] Empty/Unknown ├zram5 1.96g [252:5] Empty/Unknown ├zram6 1.96g [252:6] Empty/Unknown └zram7 1.96g [252:7] Empty/Unknown *********************************** at /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : inactive sdd[1](S) 9766308352 blocks super 1.2 unused devices: <none>
