It was this part of the original post
> The raid didn't automatically assemble so I did
> `mdadm --assemble` but really screwed up and put the 5 new disks in a
> different array
Basically, I did an `mdadm --assemble /dev/md1 <new disks for md0>`
instead of `mdadm --assemble /dev/md0 <new disks for md0>`. Further
complicated by the fact the md1 was missing a disk, so I let 1 of the
5 disks become a full member md1 since I didn't catch my error in time
and enough recovery on md1 had occurred to wipe out any data transfer
from the reshape on md0. The other 4 became hot spares. This wiped the
super block on those 5 new disks, the super blocks no longer contain
the correct information showing the original reshape attempt on md0.
I have yet to dive into the code but it seems likely that I can
manually reconstruct the appropriate super blocks for these 4 disks
that still contain valid data as a result of the reshape with a worst
case of ~1/5th data loss.
Here are the `mdadm --examine` for the 4 remaining drives destined for
md0, they believe they belong to md1 (a 6 x 4tb raid 5)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : 2e3e6281:d6735073:c92ffd33:cb6d1413
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 03:18:04 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : e87847d5 - correct
Events : 55628
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : 268fa8cf:0299a907:5f76dd12:94f78cf2
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 03:18:03 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 89708e55 - correct
Events : 55627
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : a6914f4a:14a64337:c3546c24:42930ff9
Name : any:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : d8fa02f7:8dd22cea:06d0de1d:32d4039d
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 03:37:53 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 48c66f6b - correct
Events : 55861
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : f464edaa:6338d020:7541f151:678234d2
Name : big-nas:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : f6c7b906:7aa7bbe7:aec61998:d3183973
Internal Bitmap : 8 sectors from superblock
Update Time : Thu Jan 14 13:04:45 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 24b2884e - correct
Events : 62212
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
The 5th and final 10tb disk I allowed to become the 6th member of the
6 x 4tb raid 5 after my screwup
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : f464edaa:6338d020:7541f151:678234d2
Name : big-nas:1
Creation Time : Sun Apr 6 05:23:45 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 19532619776 (9313.88 GiB 10000.70 GB)
Array Size : 19534425600 (18629.48 GiB 20003.25 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 249856 sectors
Super Offset : 8 sectors
Unused Space : before=249776 sectors, after=11718849536 sectors
State : clean
Device UUID : 527db0c9:1636b7c4:06afabaf:35d6db73
Internal Bitmap : 8 sectors from superblock
Update Time : Sat Jan 16 06:11:30 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : dcdbb6cf - correct
Events : 62885
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
On Sun, Jan 17, 2021 at 6:49 PM NeilBrown <neil@xxxxxxxxxx> wrote:
>
> On Thu, Jan 14 2021, Nathan Brown wrote:
>
> > Scenario:
> >
> > I had a 4 by 10TB raid 5 array and was adding 5 more disks and
> > reshaping it to a raid6. This was working just fine until I got a
> > little too aggressive with perf tuning and caused `mdadm` to
> > completely hang. I froze the rebuild and rebooted the server to wipe
> > away my tuning mess. The raid didn't automatically assemble so I did
> > `mdadm --assemble` but really screwed up and put the 5 new disks in a
> > different array. Not sure why the superblock on those disks didn't
> > stop `mdadm` from putting them into service but the end result was the
> > superblock on those 5 new drives got wiped. That array was missing a
>
> What does this mean "the superblock on those 5 new drives got wiped"?
> How do you think that happened.
> Can you please report the current super blocks (mdadm --examine) anyway
> please.
>
> NeilBrown
