Thanks for the extra details.
It would probably work to just add '-f' to the assemble line. Then it should
assemble the array, include the space (sdb4 currently thinks it is a spare -
not sure why), and proceed with the reshape.
The alternative is simply to re-create the array:
mdadm -C /dev/md4 -l5 -n5 -c 512 --layout ls /dev/sd{c,d,e,f,g}4 --assume-clean
Then fsck to make sure it looks OK - it should as long as the devices haven't
renamed themselves again.
Then add sdb4 as a spare and try the 'grow' again.
I'd probably try the "--assemble -f" first. It if completely fails try the
-C.
If it works - great.
If it seems to start, but doesn't progress properly (unlikely), don't try the
-C - show my the new "-E" output and we'll take it from there.
NeilBrown
On Wed, 13 Oct 2010 19:32:12 +0200
Simon SÃHIER <simon@xxxxxxxxx> wrote:
> On Wed, Oct 13, 2010 at 07:37:59PM +1100, Neil Brown wrote:
> > On Wed, 13 Oct 2010 10:18:33 +0200
> > Simon SÃHIER <simon@xxxxxxxxx> wrote:
> >
> > > On Wed, Oct 13, 2010 at 11:08:23AM +1100, Neil Brown wrote:
> > > > On Wed, 13 Oct 2010 00:59:52 +0200
> > > > Simon SÃHIER <simon@xxxxxxxxx> wrote:
> > > >
> > > > > On 12 oct. 2010 22:46:12, Neil Brown wrote :
> > > > > > On Tue, 12 Oct 2010 16:27:53 +0200
> > > > > >
> > > > > > Simon S <simon@xxxxxxxxx> wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I had a config with 5 disks and 3 raid 5 arrays:
> > > > > > >
> > > > > > > md2 : system root
> > > > > > > md3 : swap
> > > > > > > md4 : data
> > > > > > >
> > > > > > > I added a 6th disk with the intention of growing my raid5 into raid6.
> > > > > > >
> > > > > > > The step I used were :
> > > > > > >
> > > > > > > # mdadm /dev/mdX -a /dev/newdiskX
> > > > > > > # mdadm -G --level 6 -n 6 /dev/mdX --backup-file /mdXbackup
> > > > > > >
> > > > > > > (yes, with backup file on root partition md2...)
> > > > > >
> > > > > > Bad idea.. Very bad idea.
> > > > > >
> > > > > > > The md3 array reshaped without any problem.
> > > > > > > md2 seemed to reshape well until it reaches 50.4%, then the rebuild speed
> > > > > > > stalled at 14Kb/s.
> > > > > >
> > > > > > This is the expected consequence of that bad idea. Unfortunately it would
> > > > > > be hard to reliably get mdadm to complain about that, though I guess the
> > > > > > common cases are easy to protect against ... added to 'todo' list
> > > > > >
> > > > > > > md4 was still in the state "resync=DELAYED" then.
> > > > > > >
> > > > > > > As the rebuild process seemed hung, I restart the machine ... bad idea.
> > > > > >
> > > > > > Not really, nothing else would have worked.
> > > > > >
> > > > > > > Now mdadm refuses to assemble md2 and md4, and displays this message :
> > > > > > > mdadm: Failed to restore critical section for reshape, sorry.
> > > > > > >
> > > > > > > Possibly you needed to specify the --backup-file
> > > > > > >
> > > > > > > md2 is my linux installation, not very bad if I lose this one.
> > > > > > >
> > > > > > > md4 however contains valuable data.
> > > > > > >
> > > > > > > While md4 was still in the state resync=DELAYED before the shutdown, I
> > > > > > > expect it should not has been (to much) modified and can be recovered.
> > > > > >
> > > > > > Very true.
> > > > > >
> > > > > > > Any idea on how I could safely do it ?
> > > > > > >
> > > > > > > Should I give a try to the hack "Get 'Grow_restart' to always return 0."
> > > > > > > mentionned by Neil Brown on 22 april 2010 in this mailing list ?
> > > > > >
> > > > > > That is your best bet. I plan to make that easier to do in mdadm-3.2 (no
> > > > > > recompile necessary).
> > > > > >
> > > > > > Before you do, check "mdadm -E /dev/newdiskX" and make sure the "Reshape
> > > > > > position" is 0. If it is you should be fine. I
> > > > > >
> > > > > > It won't be for md2 of course. So md will quite possible have some
> > > > > > corruption. Run fsck on it an it will probably be mostly OK, but there is
> > > > > > a reasonable chance that some files will be corrupted. Whether and when
> > > > > > you will notice is impossible to guess.
> > > > >
> > > > > Thanks for your answer Neil,
> > > > >
> > > > > I recompiled mdadm 3.1.4 with return 0 in the beginning of the function
> > > > > Grow_restart (mistake was made with 3.1.2). I have one more question :
> > > > >
> > > > > I first tried assembling the least valued array, md2. It starts reshaping from
> > > > > where it stops, in the first seconds around 1300 K/s, and rapidly above 10K/s.
> > > > >
> > > > > While my backup file for md4 (the array I care about) was also on md2. Do I
> > > > > have to expect a problem assembling md4 with the modified version of mdadm, or
> > > > > can I go without worying md2 (rootfs) isn't assembled ?
> > > >
> > > > The backup file for md4 would have been essentially empty. It can be created
> > > > anew elsewhere. I probably wouldn't rick using the original backup file
> > > > even if you can access it, as it could be corrupted.
> > > > So when you assemble md4, give it a fresh backup file in some stable location,
> > > > and use the hacked mdadm.
> > > >
> > > > NeilBrown
> > > >
> > >
> > > I tried
> > >
> > > # mdadm -A --backup-file=/new-empty-md4backup-file /dev/md4
> > >
> > > but the array is now in "inactive" state with 6 spares :
> > >
> > > md4 : inactive sdc4[0](S) sdh4[6](S) sdg4[5](S) sdf4[3](S) sde4[2](S) sdd4[1](S)
> > > 1411288041 blocks super 1.2
> > >
> > > I'm a bit confuse on what I could do now.
> >
> > That surprises me a little.
> > Try:
> > mdadm -S /dev/md4
> > mdadm -Avv --backup-file=/new-empty-md4backup-file /dev/md4
> > dmesg | tail -100
> > mdadm -E /dev/sd[cd]4
> >
> > and send all of the output.
> >
> > NeilBrown
> >
>
> # mdadm -S /dev/md4
>
> mdadm: stopped /dev/md4
>
>
> # mdadm -Avv --backup-file=/new-empty-md4backup-file /dev/md4
>
> mdadm: looking for devices for /dev/md4
> mdadm: no RAID superblock on /dev/md/3
> mdadm: /dev/md/3 has wrong uuid.
> mdadm: no RAID superblock on /dev/md1
> mdadm: /dev/md1 has wrong uuid.
> mdadm: cannot open device /dev/sdg3: Device or resource busy
> mdadm: /dev/sdg3 has wrong uuid.
> mdadm: /dev/sdg2 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdg1
> mdadm: /dev/sdg1 has wrong uuid.
> mdadm: cannot open device /dev/sdg: Device or resource busy
> mdadm: /dev/sdg has wrong uuid.
> mdadm: cannot open device /dev/sdf3: Device or resource busy
> mdadm: /dev/sdf3 has wrong uuid.
> mdadm: /dev/sdf2 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdf1
> mdadm: /dev/sdf1 has wrong uuid.
> mdadm: cannot open device /dev/sdf: Device or resource busy
> mdadm: /dev/sdf has wrong uuid.
> mdadm: cannot open device /dev/sde3: Device or resource busy
> mdadm: /dev/sde3 has wrong uuid.
> mdadm: /dev/sde2 has wrong uuid.
> mdadm: cannot open device /dev/sde1: Device or resource busy
> mdadm: /dev/sde1 has wrong uuid.
> mdadm: cannot open device /dev/sde: Device or resource busy
> mdadm: /dev/sde has wrong uuid.
> mdadm: cannot open device /dev/sdd3: Device or resource busy
> mdadm: /dev/sdd3 has wrong uuid.
> mdadm: /dev/sdd2 has wrong uuid.
> mdadm: cannot open device /dev/sdd1: Device or resource busy
> mdadm: /dev/sdd1 has wrong uuid.
> mdadm: cannot open device /dev/sdd: Device or resource busy
> mdadm: /dev/sdd has wrong uuid.
> mdadm: cannot open device /dev/sdc3: Device or resource busy
> mdadm: /dev/sdc3 has wrong uuid.
> mdadm: /dev/sdc2 has wrong uuid.
> mdadm: cannot open device /dev/sdc1: Device or resource busy
> mdadm: /dev/sdc1 has wrong uuid.
> mdadm: cannot open device /dev/sdc: Device or resource busy
> mdadm: /dev/sdc has wrong uuid.
> mdadm: cannot open device /dev/sdb3: Device or resource busy
> mdadm: /dev/sdb3 has wrong uuid.
> mdadm: /dev/sdb2 has wrong uuid.
> mdadm: cannot open device /dev/sdb1: Device or resource busy
> mdadm: /dev/sdb1 has wrong uuid.
> mdadm: cannot open device /dev/sdb: Device or resource busy
> mdadm: /dev/sdb has wrong uuid.
> mdadm: cannot open device /dev/sda5: Device or resource busy
> mdadm: /dev/sda5 has wrong uuid.
> mdadm: no RAID superblock on /dev/sda2
> mdadm: /dev/sda2 has wrong uuid.
> mdadm: cannot open device /dev/sda1: Device or resource busy
> mdadm: /dev/sda1 has wrong uuid.
> mdadm: cannot open device /dev/sda: Device or resource busy
> mdadm: /dev/sda has wrong uuid.
> mdadm: /dev/sdg4 is identified as a member of /dev/md4, slot 4.
> mdadm: /dev/sdf4 is identified as a member of /dev/md4, slot 3.
> mdadm: /dev/sde4 is identified as a member of /dev/md4, slot 2.
> mdadm: /dev/sdd4 is identified as a member of /dev/md4, slot 1.
> mdadm: /dev/sdc4 is identified as a member of /dev/md4, slot 0.
> mdadm: /dev/sdb4 is identified as a member of /dev/md4, slot -1.
> mdadm:/dev/md4 has an active reshape - checking if critical section needs to be restored
> mdadm: added /dev/sdd4 to /dev/md4 as 1
> mdadm: added /dev/sde4 to /dev/md4 as 2
> mdadm: added /dev/sdf4 to /dev/md4 as 3
> mdadm: added /dev/sdg4 to /dev/md4 as 4
> mdadm: no uptodate device for slot 5 of /dev/md4
> mdadm: added /dev/sdb4 to /dev/md4 as -1
> mdadm: added /dev/sdc4 to /dev/md4 as 0
> mdadm: /dev/md4 assembled from 5 drives and 1 spare - not enough to start the array while not clean - consider --force.
>
>
> # dmesg | tail -n100
>
> [ 11.127010] HDA Intel 0000:00:1b.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
> [ 11.127038] HDA Intel 0000:00:1b.0: setting latency timer to 64
> [ 11.196601] alloc irq_desc for 16 on node -1
> [ 11.196603] alloc kstat_irqs on node -1
> [ 11.196611] pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> [ 11.196614] pci 0000:00:02.0: setting latency timer to 64
> [ 11.203560] alloc irq_desc for 32 on node -1
> [ 11.203563] alloc kstat_irqs on node -1
> [ 11.203573] pci 0000:00:02.0: irq 32 for MSI/MSI-X
> [ 11.203602] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
> [ 11.230846] Error: Driver 'pcspkr' is already registered, aborting...
> [ 11.251260] hda_codec: ALC662 rev1: BIOS auto-probing.
> [ 11.252651] input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/input/input5
> [ 11.848246] md: md2 stopped.
> [ 11.852104] md: bind<sdd2>
> [ 11.852253] md: bind<sde2>
> [ 11.852386] md: bind<sdf2>
> [ 11.852636] md: bind<sdg2>
> [ 11.852845] md: bind<sdb2>
> [ 11.852932] md: bind<sdc2>
> [ 11.882369] raid5: reshape will continue
> [ 11.882378] raid5: device sdc2 operational as raid disk 0
> [ 11.882380] raid5: device sdg2 operational as raid disk 4
> [ 11.882382] raid5: device sdf2 operational as raid disk 3
> [ 11.882383] raid5: device sde2 operational as raid disk 2
> [ 11.882385] raid5: device sdd2 operational as raid disk 1
> [ 11.882767] raid5: allocated 6386kB for md2
> [ 11.882797] 0: w=1 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
> [ 11.882799] 5: w=1 pa=18 pr=6 m=2 a=2 r=6 op1=1 op2=0
> [ 11.882801] 4: w=2 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
> [ 11.882803] 3: w=3 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
> [ 11.882805] 2: w=4 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
> [ 11.882807] 1: w=5 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
> [ 11.882809] raid5: raid level 6 set md2 active with 5 out of 6 devices, algorithm 2
> [ 11.882849] RAID5 conf printout:
> [ 11.882851] --- rd:6 wd:5
> [ 11.882852] disk 0, o:1, dev:sdc2
> [ 11.882854] disk 1, o:1, dev:sdd2
> [ 11.882855] disk 2, o:1, dev:sde2
> [ 11.882856] disk 3, o:1, dev:sdf2
> [ 11.882858] disk 4, o:1, dev:sdg2
> [ 11.882859] disk 5, o:1, dev:sdb2
> [ 11.882860] ...ok start reshape thread
> [ 11.882905] md2: detected capacity change from 0 to 34376515584
> [ 11.882970] md: md2 switched to read-write mode.
> [ 11.883452] md: reshape of RAID array md2
> [ 11.883453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 11.883455] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
> [ 11.883459] md: using 128k window, over a total of 8392704 blocks.
> [ 11.954838] md2: unknown partition table
> [ 12.939843] Adding 1648632k swap on /dev/sda5. Priority:-1 extents:1 across:1648632k
> [ 13.142646] EXT3 FS on sda1, internal journal
> [ 13.255820] loop: module loaded
> [ 100.224309] e1000e 0000:00:19.0: irq 30 for MSI/MSI-X
> [ 100.280126] e1000e 0000:00:19.0: irq 30 for MSI/MSI-X
> [ 100.280456] ADDRCONF(NETDEV_UP): eth2: link is not ready
> [ 104.940830] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> [ 104.941081] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
> [ 143.313242] md: md_do_sync() got signal ... exiting
> [ 143.373125] md: md2 stopped.
> [ 143.373138] md: unbind<sdc2>
> [ 143.384082] md: export_rdev(sdc2)
> [ 143.384113] md: unbind<sdb2>
> [ 143.400081] md: export_rdev(sdb2)
> [ 143.400107] md: unbind<sdg2>
> [ 143.416081] md: export_rdev(sdg2)
> [ 143.416108] md: unbind<sdf2>
> [ 143.432080] md: export_rdev(sdf2)
> [ 143.432105] md: unbind<sde2>
> [ 143.448080] md: export_rdev(sde2)
> [ 143.448104] md: unbind<sdd2>
> [ 143.464081] md: export_rdev(sdd2)
> [ 143.464405] md2: detected capacity change from 34376515584 to 0
> [ 252.687538] md: md4 stopped.
> [ 252.690104] md: bind<sdd4>
> [ 252.690266] md: bind<sde4>
> [ 252.690415] md: bind<sdf4>
> [ 252.696210] md: bind<sdg4>
> [ 252.718353] md: bind<sdb4>
> [ 252.723594] md: bind<sdc4>
> [ 332.729180] md: md4 stopped.
> [ 332.729190] md: unbind<sdc4>
> [ 332.740090] md: export_rdev(sdc4)
> [ 332.740165] md: unbind<sdb4>
> [ 332.752030] md: export_rdev(sdb4)
> [ 332.752092] md: unbind<sdg4>
> [ 332.768081] md: export_rdev(sdg4)
> [ 332.768140] md: unbind<sdf4>
> [ 332.784081] md: export_rdev(sdf4)
> [ 332.784139] md: unbind<sde4>
> [ 332.800081] md: export_rdev(sde4)
> [ 332.800141] md: unbind<sdd4>
> [ 332.816089] md: export_rdev(sdd4)
> [ 556.983627] md: md4 stopped.
> [ 556.988921] md: bind<sdd4>
> [ 556.989094] md: bind<sde4>
> [ 556.989239] md: bind<sdf4>
> [ 556.989391] md: bind<sdg4>
> [ 556.989642] md: bind<sdb4>
> [ 556.989787] md: bind<sdc4>
>
>
> # mdadm -E /dev/sd[cd]4
>
> /dev/sdc4:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : ec600d5d:00cc3fc7:862a4878:9d191184
> Name : citrouille:4
> Creation Time : Wed Sep 15 17:28:55 2010
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 470429347 (224.32 GiB 240.86 GB)
> Array Size : 1881714688 (897.27 GiB 963.44 GB)
> Used Dev Size : 470428672 (224.32 GiB 240.86 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : e8ef9525:cfb44c96:b5e209ea:b307c619
>
> Reshape pos'n : 0
> New Layout : left-symmetric
>
> Update Time : Tue Oct 12 00:01:03 2010
> Checksum : 6dfe6f7b - correct
> Events : 97
>
> Layout : left-symmetric-6
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAAAA. ('A' == active, '.' == missing)
> /dev/sdd4:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x4
> Array UUID : ec600d5d:00cc3fc7:862a4878:9d191184
> Name : citrouille:4
> Creation Time : Wed Sep 15 17:28:55 2010
> Raid Level : raid6
> Raid Devices : 6
>
> Avail Dev Size : 470429347 (224.32 GiB 240.86 GB)
> Array Size : 1881714688 (897.27 GiB 963.44 GB)
> Used Dev Size : 470428672 (224.32 GiB 240.86 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : 08a661f3:eace7b1f:26fd20a8:ac0ae049
>
> Reshape pos'n : 0
> New Layout : left-symmetric
>
> Update Time : Tue Oct 12 00:01:03 2010
> Checksum : b32a5d21 - correct
> Events : 97
>
> Layout : left-symmetric-6
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAAA. ('A' == active, '.' == missing)
>
>
>
> # mdadm -V
> mdadm - v3.1.4 - 31st August 2010 - with Grow_restart always 0
> (hacked mdadm)
>
> # uname -a
> Linux citrouillerescue 2.6.32-5-amd64 #1 SMP Fri Sep 17 21:50:19 UTC 2010 x86_64 GNU/Linux
>
> # cat /etc/issue.net
> Debian GNU/Linux squeeze/sid
>
> Just in case, I posted the full output of mdadm -E /dev/sd?4 here : http://pastebin.com/zV6s2Npi
>
> Hope it helps.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
