When I looked at your detailed files you sent a few days ago, all of the reshapes (on all disks) indicated that they were at position 0, so it kind of appears that the reshape never actually started at all and hung immediately which is probably why it cannot find the critical section, it hung prior to that getting done. Not entirely sure how to undo a reshape that failed like this. On Mon, Apr 6, 2020 at 10:29 AM Alexander Shenkin <al@xxxxxxxxxxx> wrote: > > > > On 4/4/2020 9:19 AM, Alexander Shenkin wrote: > > On 4/1/2020 11:16 AM, Alexander Shenkin wrote: > >> Hi all, > >> > >> I had a problem that caused my Ubuntu Server 14 to go down, and it now > >> will not boot. I added a drive to my raid1 (/dev/md0 -> /boot) + raid6 > >> (/dev/md2 -> /) setup. I was previously running with /dev/md0 (raid1) > >> assembled from /dev/sd[a-f]1, and /dev/md2 (raid6) assembled > >> /dev/sd[a-f]3. Both partitions from the new drive were successfully > >> added to the two md arrays, and the raid1 (the smaller of the two) > >> resync'd successfully it seemed (I think that resync is the correct word > >> to use, meaning that mdadm is spreading out the data amongst the drives > >> once an array has been grown). When resyncing the larger raid6, > >> however, the sync speed was quite slow (kb's/sec), and got slower and > >> slower (7kb/sec... 5kb/sec... 3kb/sec... 1kb/sec...) until the system > >> entirely halted and I eventually turned the power off. It now will not > >> boot. > >> > >> Thanks to Roger Heflin's help, I've booted into a livecd environment > >> (ubuntu server 18) with all the necessary raid personalities available. > >> > >> When trying to --assemble md127 (raid6), i get the following error: > >> "Failure to restore critical section for reshape, sorry. Perhaps you > >> needed to specify the --backup-file". Needless to say, I didn't save a > >> backup file when adding the drive. > >> > >> I have attached some diagnostic output here. It seems that my raid6 > >> array is not being recognized as such. I'm not sure what the next steps > >> are - do I need to figure out how to get the resync up and running > >> again? Any help would be greatly appreciated. > >> > >> Many thanks, > >> > >> Allie > >> > > > > Hello again, > > > > Reading through https://raid.wiki.kernel.org/index.php/Assemble_Run and > > https://raid.wiki.kernel.org/index.php/RAID_Recovery, the advice seems > > to be to force assemble the array given that my event counts are all the > > same. However, I don't want to do that until some experts have chimed > > in. I've seen some other threads about using overlays to avoid data > > loss... not sure if that is still a recommended method. > > > > I'm including the dmesg and mdadm --examine output here... any advice > > much appreciated! > > > > Thanks, > > Allie > > > > (note below - arrays are on /dev/md[a-g]) > > > > root@ubuntu-server:/home/ubuntu-server# mdadm -A --scan --verbose > > mdadm: looking for devices for further assembly > > mdadm: Cannot assemble mbr metadata on /dev/sdh1 > > mdadm: Cannot assemble mbr metadata on /dev/sdh > > mdadm: no recogniseable superblock on /dev/md/bobbiflekman:0 > > mdadm: no recogniseable superblock on /dev/sdf4 > > mdadm: /dev/sdf3 is busy - skipping > > mdadm: no recogniseable superblock on /dev/sdf2 > > mdadm: /dev/sdf1 is busy - skipping > > mdadm: Cannot assemble mbr metadata on /dev/sdf > > mdadm: no recogniseable superblock on /dev/sdg4 > > mdadm: /dev/sdg3 is busy - skipping > > mdadm: no recogniseable superblock on /dev/sdg2 > > mdadm: /dev/sdg1 is busy - skipping > > mdadm: Cannot assemble mbr metadata on /dev/sdg > > mdadm: no recogniseable superblock on /dev/sde4 > > mdadm: /dev/sde3 is busy - skipping > > mdadm: no recogniseable superblock on /dev/sde2 > > mdadm: /dev/sde1 is busy - skipping > > mdadm: Cannot assemble mbr metadata on /dev/sde > > mdadm: cannot open device /dev/sr1: No medium found > > mdadm: cannot open device /dev/sr0: No medium found > > mdadm: no recogniseable superblock on /dev/sdd4 > > mdadm: /dev/sdd3 is busy - skipping > > mdadm: no recogniseable superblock on /dev/sdd2 > > mdadm: /dev/sdd1 is busy - skipping > > mdadm: Cannot assemble mbr metadata on /dev/sdd > > mdadm: no recogniseable superblock on /dev/sdc4 > > mdadm: /dev/sdc3 is busy - skipping > > mdadm: no recogniseable superblock on /dev/sdc2 > > mdadm: /dev/sdc1 is busy - skipping > > mdadm: Cannot assemble mbr metadata on /dev/sdc > > mdadm: no recogniseable superblock on /dev/sdb4 > > mdadm: /dev/sdb3 is busy - skipping > > mdadm: no recogniseable superblock on /dev/sdb2 > > mdadm: /dev/sdb1 is busy - skipping > > mdadm: Cannot assemble mbr metadata on /dev/sdb > > mdadm: no recogniseable superblock on /dev/sda4 > > mdadm: /dev/sda3 is busy - skipping > > mdadm: no recogniseable superblock on /dev/sda2 > > mdadm: /dev/sda1 is busy - skipping > > mdadm: Cannot assemble mbr metadata on /dev/sda > > mdadm: no recogniseable superblock on /dev/loop7 > > mdadm: no recogniseable superblock on /dev/loop6 > > mdadm: no recogniseable superblock on /dev/loop5 > > mdadm: no recogniseable superblock on /dev/loop4 > > mdadm: no recogniseable superblock on /dev/loop3 > > mdadm: no recogniseable superblock on /dev/loop2 > > mdadm: no recogniseable superblock on /dev/loop1 > > mdadm: no recogniseable superblock on /dev/loop0 > > mdadm: No arrays found in config file or automatically > > > > Hi again all, > > Apologies for all the self-followed-up emails. I realized my previous > --examine was done without stopping the array in question. When stopped > and reexamined, the critical bit seems to be this: > > mdadm: /dev/sdf3 is identified as a member of /dev/md/ubuntu:2, slot 4. > mdadm: /dev/sdg3 is identified as a member of /dev/md/ubuntu:2, slot 6. > mdadm: /dev/sde3 is identified as a member of /dev/md/ubuntu:2, slot 5. > mdadm: /dev/sdd3 is identified as a member of /dev/md/ubuntu:2, slot 3. > mdadm: /dev/sdc3 is identified as a member of /dev/md/ubuntu:2, slot 2. > mdadm: /dev/sdb3 is identified as a member of /dev/md/ubuntu:2, slot 1. > mdadm: /dev/sda3 is identified as a member of /dev/md/ubuntu:2, slot 0. > mdadm: /dev/md/ubuntu:2 has an active reshape - checking if critical > section needs to be restored > mdadm: No backup metadata on device-6 > mdadm: Failed to find backup of critical section > mdadm: Failed to restore critical section for reshape, sorry. > Possibly you needed to specify the --backup-file > > That is, it thinks there is an active reshape. > > Thanks, > Allie > > > root@ubuntu-server:/home/ubuntu-server# mdadm -A --scan --verbose > mdadm: looking for devices for further assembly > mdadm: Cannot assemble mbr metadata on /dev/sdh1 > mdadm: Cannot assemble mbr metadata on /dev/sdh > mdadm: no recogniseable superblock on /dev/md/bobbiflekman:0 > mdadm: no recogniseable superblock on /dev/sdf4 > mdadm: No super block found on /dev/sdf2 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdf2 > mdadm: /dev/sdf1 is busy - skipping > mdadm: No super block found on /dev/sdf (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdf > mdadm: No super block found on /dev/sdg4 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdg4 > mdadm: No super block found on /dev/sdg2 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdg2 > mdadm: /dev/sdg1 is busy - skipping > mdadm: No super block found on /dev/sdg (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdg > mdadm: No super block found on /dev/sde4 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sde4 > mdadm: No super block found on /dev/sde2 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sde2 > mdadm: /dev/sde1 is busy - skipping > mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sde > mdadm: cannot open device /dev/sr1: No medium found > mdadm: cannot open device /dev/sr0: No medium found > mdadm: No super block found on /dev/sdd4 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdd4 > mdadm: No super block found on /dev/sdd2 (Expected magic a92b4efc, got > 9c6196f1) > mdadm: no RAID superblock on /dev/sdd2 > mdadm: /dev/sdd1 is busy - skipping > mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdd > mdadm: No super block found on /dev/sdc4 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdc4 > mdadm: No super block found on /dev/sdc2 (Expected magic a92b4efc, got > 9c6196f1) > mdadm: no RAID superblock on /dev/sdc2 > mdadm: /dev/sdc1 is busy - skipping > mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdc > mdadm: No super block found on /dev/sdb4 (Expected magic a92b4efc, got > 53425553) > mdadm: no RAID superblock on /dev/sdb4 > mdadm: No super block found on /dev/sdb2 (Expected magic a92b4efc, got > 9c6196f1) > mdadm: no RAID superblock on /dev/sdb2 > mdadm: /dev/sdb1 is busy - skipping > mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sdb > mdadm: No super block found on /dev/sda4 (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sda4 > mdadm: No super block found on /dev/sda2 (Expected magic a92b4efc, got > 9c6196f1) > mdadm: no RAID superblock on /dev/sda2 > mdadm: /dev/sda1 is busy - skipping > mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got > 00000000) > mdadm: no RAID superblock on /dev/sda > mdadm: No super block found on /dev/loop7 (Expected magic a92b4efc, got > 72756769) > mdadm: no RAID superblock on /dev/loop7 > mdadm: No super block found on /dev/loop6 (Expected magic a92b4efc, got > 69622f21) > mdadm: no RAID superblock on /dev/loop6 > mdadm: No super block found on /dev/loop5 (Expected magic a92b4efc, got > 14ea0a05) > mdadm: no RAID superblock on /dev/loop5 > mdadm: No super block found on /dev/loop4 (Expected magic a92b4efc, got > 1824ef5d) > mdadm: no RAID superblock on /dev/loop4 > mdadm: No super block found on /dev/loop3 (Expected magic a92b4efc, got > 1e993ae9) > mdadm: no RAID superblock on /dev/loop3 > mdadm: No super block found on /dev/loop2 (Expected magic a92b4efc, got > cb4d8c8e) > mdadm: no RAID superblock on /dev/loop2 > mdadm: No super block found on /dev/loop1 (Expected magic a92b4efc, got > d2964063) > mdadm: no RAID superblock on /dev/loop1 > mdadm: No super block found on /dev/loop0 (Expected magic a92b4efc, got > e7e108a6) > mdadm: no RAID superblock on /dev/loop0 > mdadm: /dev/sdf3 is identified as a member of /dev/md/ubuntu:2, slot 4. > mdadm: /dev/sdg3 is identified as a member of /dev/md/ubuntu:2, slot 6. > mdadm: /dev/sde3 is identified as a member of /dev/md/ubuntu:2, slot 5. > mdadm: /dev/sdd3 is identified as a member of /dev/md/ubuntu:2, slot 3. > mdadm: /dev/sdc3 is identified as a member of /dev/md/ubuntu:2, slot 2. > mdadm: /dev/sdb3 is identified as a member of /dev/md/ubuntu:2, slot 1. > mdadm: /dev/sda3 is identified as a member of /dev/md/ubuntu:2, slot 0. > mdadm: /dev/md/ubuntu:2 has an active reshape - checking if critical > section needs to be restored > mdadm: No backup metadata on device-6 > mdadm: Failed to find backup of critical section > mdadm: Failed to restore critical section for reshape, sorry. > Possibly you needed to specify the --backup-file > mdadm: looking for devices for further assembly > mdadm: /dev/sdf1 is busy - skipping > mdadm: /dev/sdg1 is busy - skipping > mdadm: /dev/sde1 is busy - skipping > mdadm: /dev/sdd1 is busy - skipping > mdadm: /dev/sdc1 is busy - skipping > mdadm: /dev/sdb1 is busy - skipping > mdadm: /dev/sda1 is busy - skipping > mdadm: No arrays found in config file or automatically
