On Sun, 13 Nov 2011 09:53:52 -0500 Travis Brown <teb@xxxxxxxxxx> wrote: > So I have the RAID rebuilding and the the output of /proc/mdstat look ok (I think): > > root@bravo:~# cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] > md126 : active raid6 sde3[0] sdd3[4] sdf3[3] sdb3[1] > 5856728064 blocks super 0.91 level 6, 512k chunk, algorithm 18 [5/3] [UU_U_] > [>....................] reshape = 2.0% (39448064/1952242688) finish=7375.5min speed=4321K/sec > > md11 : active raid6 sde2[0] sdb2[1] sdd2[4] sdc2[2] sdf2[3] > 3180672 blocks level 6, 64k chunk, algorithm 2 [5/5] [UUUUU] > > md10 : active raid1 sde1[0] sdb1[1] sdd1[4] sdf1[3] sdc1[2] > 208704 blocks [5/5] [UUUUU] > > unused devices: <none> > > > > But mdadm --detail /dev/md126 now shows I have a drive removed: That is unfortunate. It looks like sdc3 didn't make it back into the array when you re-assembled it. I wonder why not? Any kernel logs from when you started the array. Your RAID6 is now double-degraded so another failure would be bad. Should be unlikely though. As soon as the reshape finished you need to see about adding sdc3 back in to the array so that it rebuilds. The rebuild will be a lot faster than the reshape is. Best thing to do now is to just leave it reshaping. However if there are some really important files that you could conveniently back up, now might be a good time, just in case. NeilBrown > > /dev/md126: > Version : 0.91 > Creation Time : Wed Nov 10 20:19:03 2010 > Raid Level : raid6 > Array Size : 5856728064 (5585.41 GiB 5997.29 GB) > Used Dev Size : 1952242688 (1861.80 GiB 1999.10 GB) > Raid Devices : 5 > Total Devices : 4 > Preferred Minor : 126 > Persistence : Superblock is persistent > > Update Time : Sun Nov 13 09:50:52 2011 > State : active, degraded, recovering > Active Devices : 3 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 1 > > Layout : left-symmetric-6 > Chunk Size : 512K > > Reshape Status : 2% complete > New Layout : left-symmetric > > UUID : 3fd8b303:7727aa3b:c5d110f2:f9137e1d > Events : 0.172355 > > Number Major Minor RaidDevice State > 0 8 67 0 active sync /dev/sde3 > 1 8 19 1 active sync /dev/sdb3 > 2 0 0 2 removed > 3 8 83 3 active sync /dev/sdf3 > 4 8 51 4 spare rebuilding /dev/sdd3 > > Is that expected? > > Thanks again. I am pretty confident all my data is there, and I do have a (1-day old) backup of the important stuff, but the other 2.5TB of stuff isn't really /that/ important, but I don't want to have to explain to the wife why all her favorite episodes of NCIS that she recorded are gone :) > > Thanks, > Travis > > On Nov 12, 2011, at 10:35 PM, NeilBrown wrote: > > > On Sat, 12 Nov 2011 21:56:56 -0500 Travis Brown <teb@xxxxxxxxxx> wrote: > > > >> I was reshaping my 5 drive raid 5 with spare to a raid 6 array when the drive I was using for my backup went offline. If that's not murphy's law, I don't know what is. The array is still up and usable, but I'm afraid to reboot or doing anything to it, really. Suggestions on getting this thing back to usable are very welcome. > >> > >> Thanks, > >> Travis > >> > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] > >> md126 : active raid6 sde3[0] sdf3[3] sdb3[1] sdd3[4] sdc3[2] > >> 5856728064 blocks super 0.91 level 6, 512k chunk, algorithm 18 [5/4] [UUUU_] > >> [>....................] reshape = 0.9% (19267584/1952242688) finish=623878.3min speed=51K/sec > > > > 1/ Don't Panic. > > You seem to have achieved this step quite effectively - congratulations. > > > > 2/ Stop the array cleanly. Not having a backup will only cause possible > > corruption if the machine crashes while the reshape is happening. The > > reshape has stopped so there is no chance for corruption. But you still > > need to cleanly stop the array. > > (A subsequent version of mdadm may allow you to continue the reshape > > without the stop/restart step, but we aren't there yet). > > > > 3/ Make sure you have a version of mdadm which is at least 3.2. I would > > suggest the latest:3.2.2. You particularly need the --invalid-backup > > flag. > > > > 4/ Reassemble the array with e.g. > > > > mdadm --assemble /dev/md126 --backup=/some/file \ > > --invalid-backup /dev/sd[bcdef]3 > > > > > > The backup file does not need to exist (I think). Maybe create an empty > > file and use that just to be safe. > > The "--invalid-backup" flag says to mdadm "Yes, I know the backup file is > > currently invalid and you cannot restore anything from it. I happen to > > know that there is no need to restore anything because I did a clean > > shutdown. Just use the backup file for making new backups as you continue > > the reshape". > > > > NeilBrown > > > > > >> > >> /dev/md126: > >> Version : 0.91 > >> Creation Time : Wed Nov 10 20:19:03 2010 > >> Raid Level : raid6 > >> Array Size : 5856728064 (5585.41 GiB 5997.29 GB) > >> Used Dev Size : 1952242688 (1861.80 GiB 1999.10 GB) > >> Raid Devices : 5 > >> Total Devices : 5 > >> Preferred Minor : 126 > >> Persistence : Superblock is persistent > >> > >> Update Time : Sat Nov 12 21:55:46 2011 > >> State : clean, degraded, recovering > >> Active Devices : 4 > >> Working Devices : 5 > >> Failed Devices : 0 > >> Spare Devices : 1 > >> > >> Layout : left-symmetric-6 > >> Chunk Size : 512K > >> > >> Reshape Status : 0% complete > >> New Layout : left-symmetric > >> > >> UUID : 3fd8b303:7727aa3b:c5d110f2:f9137e1d > >> Events : 0.124051 > >> > >> Number Major Minor RaidDevice State > >> 0 8 67 0 active sync /dev/sde3 > >> 1 8 19 1 active sync /dev/sdb3 > >> 2 8 35 2 active sync /dev/sdc3 > >> 3 8 83 3 active sync /dev/sdf3 > >> 4 8 51 4 spare rebuilding /dev/sdd3-- > >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in > >> the body of a message to majordomo@xxxxxxxxxxxxxxx > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html
Attachment:
signature.asc
Description: PGP signature
