On Apr 2, 2013, at 10:20 AM, Barrett Lewis <barrett.lewis.mitsi@xxxxxxxxx> wrote: > I was reshaping a 5x2tb raid5 to a 6x2tb raid6. Not knowing that > ubuntu deletes the /tmp/ folder each reboot, I specified my > --backup-file as /tmp/raid-backup.bak (this is not part of the array). > At 15.1% the system hung sufficiently that REISUB and the reset > button were ignored and I had to hold the power button down to reset > the server. After booting back from the crash, the array would not > start, and ubuntu had deleted the backup file (and everything else in > /tmp). > > The superblock already says it's raid6, all members are present and > the event counters are the same on all disks. I tried > > ubuntu@ubuntu:~$ sudo mdadm --assemble --force --run --verbose > /dev/md0 /dev/sd[abcdef] > mdadm: looking for devices for /dev/md0 > mdadm: /dev/sda is identified as a member of /dev/md0, slot 4. > mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0. > mdadm: /dev/sdc is identified as a member of /dev/md0, slot 5. > mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2. > mdadm: /dev/sde is identified as a member of /dev/md0, slot 3. > mdadm: /dev/sdf is identified as a member of /dev/md0, slot 1. > mdadm:/dev/md0 has an active reshape - checking if critical section > needs to be restored > mdadm: Failed to find backup of critical section > mdadm: Failed to restore critical section for reshape, sorry. > Possibly you needed to specify the --backup-file > > > My understanding is that the backup file is only for some early > critical part of the reshape and that it isn’t even used after that. > 15% into 8tb is well over a terrabyte so wouldn’t that be far past any > filesystem metadata? So what exactly is implied (about the state of > the reshape) by the fact that programmatically it is still requiring > the backup file? > > I have read the manpage on the --invalid-backup command but I didn't > clearly get "use it here, not here" type of information. I have the > OS drive (with deleted /tmp/raid-backup.bak) in a data recovery > process. If I actually get the backup file recovered, it could > potentially have corrupted bits. Is the best course of action to: > Supply the (potentially corrupted, but maybe some percent ok) > recovered backup file as the legitimate backup file (without > --invalid-backup)? (could this be worse than --invalid-backup and a > blank file?) > Supply the (potentially corrupted) recovered backup file WITH --invalid-backup? > Supply --invalid-backup and an empty file? > > Or if I am on the wrong path, let me know of any other thoughts or > suggestions you might have. > > If I get nothing useful back from data recovery, and I have to supply > --invalid-backup with a blank file, considering the reshape made it to > 15%, how much chance is there that the array could assemble and resume > reshape? I would gladly accept the corruption of some files vs losing > the whole file system (obviously). > > Thanks! > -- > I'm not sure about how exactly a reshape from 5-6 with no capacity change works... it is possible that it works like a shrink in that the critical section is at the end, did you try giving it a backup path with nothing there? Otherwise, I would suggest making copies of all your devices and working with the copy... when I had a similar problem, I copied my member disks into lvms then worked with snapshots so that I could undo any changes that had problems. Once I knew exactly what happened I then went back to the original disks and corrected the problem while retaining my backup in case of further problems. Also look in your recovered backup - is there actually data in it? Sam -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
