On Wed, Oct 13, 2010 at 07:37:59PM +1100, Neil Brown wrote:
> On Wed, 13 Oct 2010 10:18:33 +0200
> Simon SÉHIER <simon@xxxxxxxxx> wrote:
>
> > On Wed, Oct 13, 2010 at 11:08:23AM +1100, Neil Brown wrote:
> > > On Wed, 13 Oct 2010 00:59:52 +0200
> > > Simon SÉHIER <simon@xxxxxxxxx> wrote:
> > >
> > > > On 12 oct. 2010 22:46:12, Neil Brown wrote :
> > > > > On Tue, 12 Oct 2010 16:27:53 +0200
> > > > >
> > > > > Simon S <simon@xxxxxxxxx> wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I had a config with 5 disks and 3 raid 5 arrays:
> > > > > >
> > > > > > md2 : system root
> > > > > > md3 : swap
> > > > > > md4 : data
> > > > > >
> > > > > > I added a 6th disk with the intention of growing my raid5 into raid6.
> > > > > >
> > > > > > The step I used were :
> > > > > >
> > > > > > # mdadm /dev/mdX -a /dev/newdiskX
> > > > > > # mdadm -G --level 6 -n 6 /dev/mdX --backup-file /mdXbackup
> > > > > >
> > > > > > (yes, with backup file on root partition md2...)
> > > > >
> > > > > Bad idea.. Very bad idea.
> > > > >
> > > > > > The md3 array reshaped without any problem.
> > > > > > md2 seemed to reshape well until it reaches 50.4%, then the rebuild speed
> > > > > > stalled at 14Kb/s.
> > > > >
> > > > > This is the expected consequence of that bad idea. Unfortunately it would
> > > > > be hard to reliably get mdadm to complain about that, though I guess the
> > > > > common cases are easy to protect against ... added to 'todo' list
> > > > >
> > > > > > md4 was still in the state "resync=DELAYED" then.
> > > > > >
> > > > > > As the rebuild process seemed hung, I restart the machine ... bad idea.
> > > > >
> > > > > Not really, nothing else would have worked.
> > > > >
> > > > > > Now mdadm refuses to assemble md2 and md4, and displays this message :
> > > > > > mdadm: Failed to restore critical section for reshape, sorry.
> > > > > >
> > > > > > Possibly you needed to specify the --backup-file
> > > > > >
> > > > > > md2 is my linux installation, not very bad if I lose this one.
> > > > > >
> > > > > > md4 however contains valuable data.
> > > > > >
> > > > > > While md4 was still in the state resync=DELAYED before the shutdown, I
> > > > > > expect it should not has been (to much) modified and can be recovered.
> > > > >
> > > > > Very true.
> > > > >
> > > > > > Any idea on how I could safely do it ?
> > > > > >
> > > > > > Should I give a try to the hack "Get 'Grow_restart' to always return 0."
> > > > > > mentionned by Neil Brown on 22 april 2010 in this mailing list ?
> > > > >
> > > > > That is your best bet. I plan to make that easier to do in mdadm-3.2 (no
> > > > > recompile necessary).
> > > > >
> > > > > Before you do, check "mdadm -E /dev/newdiskX" and make sure the "Reshape
> > > > > position" is 0. If it is you should be fine. I
> > > > >
> > > > > It won't be for md2 of course. So md will quite possible have some
> > > > > corruption. Run fsck on it an it will probably be mostly OK, but there is
> > > > > a reasonable chance that some files will be corrupted. Whether and when
> > > > > you will notice is impossible to guess.
> > > >
> > > > Thanks for your answer Neil,
> > > >
> > > > I recompiled mdadm 3.1.4 with return 0 in the beginning of the function
> > > > Grow_restart (mistake was made with 3.1.2). I have one more question :
> > > >
> > > > I first tried assembling the least valued array, md2. It starts reshaping from
> > > > where it stops, in the first seconds around 1300 K/s, and rapidly above 10K/s.
> > > >
> > > > While my backup file for md4 (the array I care about) was also on md2. Do I
> > > > have to expect a problem assembling md4 with the modified version of mdadm, or
> > > > can I go without worying md2 (rootfs) isn't assembled ?
> > >
> > > The backup file for md4 would have been essentially empty. It can be created
> > > anew elsewhere. I probably wouldn't rick using the original backup file
> > > even if you can access it, as it could be corrupted.
> > > So when you assemble md4, give it a fresh backup file in some stable location,
> > > and use the hacked mdadm.
> > >
> > > NeilBrown
> > >
> >
> > I tried
> >
> > # mdadm -A --backup-file=/new-empty-md4backup-file /dev/md4
> >
> > but the array is now in "inactive" state with 6 spares :
> >
> > md4 : inactive sdc4[0](S) sdh4[6](S) sdg4[5](S) sdf4[3](S) sde4[2](S) sdd4[1](S)
> > 1411288041 blocks super 1.2
> >
> > I'm a bit confuse on what I could do now.
>
> That surprises me a little.
> Try:
> mdadm -S /dev/md4
> mdadm -Avv --backup-file=/new-empty-md4backup-file /dev/md4
> dmesg | tail -100
> mdadm -E /dev/sd[cd]4
>
> and send all of the output.
>
> NeilBrown
>
# mdadm -S /dev/md4
mdadm: stopped /dev/md4
# mdadm -Avv --backup-file=/new-empty-md4backup-file /dev/md4
mdadm: looking for devices for /dev/md4
mdadm: no RAID superblock on /dev/md/3
mdadm: /dev/md/3 has wrong uuid.
mdadm: no RAID superblock on /dev/md1
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/sdg3: Device or resource busy
mdadm: /dev/sdg3 has wrong uuid.
mdadm: /dev/sdg2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg1
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: /dev/sdf3 has wrong uuid.
mdadm: /dev/sdf2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf1
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sde3: Device or resource busy
mdadm: /dev/sde3 has wrong uuid.
mdadm: /dev/sde2 has wrong uuid.
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd3: Device or resource busy
mdadm: /dev/sdd3 has wrong uuid.
mdadm: /dev/sdd2 has wrong uuid.
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: /dev/sdd1 has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc3: Device or resource busy
mdadm: /dev/sdc3 has wrong uuid.
mdadm: /dev/sdc2 has wrong uuid.
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sdb3: Device or resource busy
mdadm: /dev/sdb3 has wrong uuid.
mdadm: /dev/sdb2 has wrong uuid.
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda5: Device or resource busy
mdadm: /dev/sda5 has wrong uuid.
mdadm: no RAID superblock on /dev/sda2
mdadm: /dev/sda2 has wrong uuid.
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sdg4 is identified as a member of /dev/md4, slot 4.
mdadm: /dev/sdf4 is identified as a member of /dev/md4, slot 3.
mdadm: /dev/sde4 is identified as a member of /dev/md4, slot 2.
mdadm: /dev/sdd4 is identified as a member of /dev/md4, slot 1.
mdadm: /dev/sdc4 is identified as a member of /dev/md4, slot 0.
mdadm: /dev/sdb4 is identified as a member of /dev/md4, slot -1.
mdadm:/dev/md4 has an active reshape - checking if critical section needs to be restored
mdadm: added /dev/sdd4 to /dev/md4 as 1
mdadm: added /dev/sde4 to /dev/md4 as 2
mdadm: added /dev/sdf4 to /dev/md4 as 3
mdadm: added /dev/sdg4 to /dev/md4 as 4
mdadm: no uptodate device for slot 5 of /dev/md4
mdadm: added /dev/sdb4 to /dev/md4 as -1
mdadm: added /dev/sdc4 to /dev/md4 as 0
mdadm: /dev/md4 assembled from 5 drives and 1 spare - not enough to start the array while not clean - consider --force.
# dmesg | tail -n100
[ 11.127010] HDA Intel 0000:00:1b.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
[ 11.127038] HDA Intel 0000:00:1b.0: setting latency timer to 64
[ 11.196601] alloc irq_desc for 16 on node -1
[ 11.196603] alloc kstat_irqs on node -1
[ 11.196611] pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 11.196614] pci 0000:00:02.0: setting latency timer to 64
[ 11.203560] alloc irq_desc for 32 on node -1
[ 11.203563] alloc kstat_irqs on node -1
[ 11.203573] pci 0000:00:02.0: irq 32 for MSI/MSI-X
[ 11.203602] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[ 11.230846] Error: Driver 'pcspkr' is already registered, aborting...
[ 11.251260] hda_codec: ALC662 rev1: BIOS auto-probing.
[ 11.252651] input: HDA Digital PCBeep as /devices/pci0000:00/0000:00:1b.0/input/input5
[ 11.848246] md: md2 stopped.
[ 11.852104] md: bind<sdd2>
[ 11.852253] md: bind<sde2>
[ 11.852386] md: bind<sdf2>
[ 11.852636] md: bind<sdg2>
[ 11.852845] md: bind<sdb2>
[ 11.852932] md: bind<sdc2>
[ 11.882369] raid5: reshape will continue
[ 11.882378] raid5: device sdc2 operational as raid disk 0
[ 11.882380] raid5: device sdg2 operational as raid disk 4
[ 11.882382] raid5: device sdf2 operational as raid disk 3
[ 11.882383] raid5: device sde2 operational as raid disk 2
[ 11.882385] raid5: device sdd2 operational as raid disk 1
[ 11.882767] raid5: allocated 6386kB for md2
[ 11.882797] 0: w=1 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[ 11.882799] 5: w=1 pa=18 pr=6 m=2 a=2 r=6 op1=1 op2=0
[ 11.882801] 4: w=2 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[ 11.882803] 3: w=3 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[ 11.882805] 2: w=4 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[ 11.882807] 1: w=5 pa=18 pr=6 m=2 a=2 r=6 op1=0 op2=0
[ 11.882809] raid5: raid level 6 set md2 active with 5 out of 6 devices, algorithm 2
[ 11.882849] RAID5 conf printout:
[ 11.882851] --- rd:6 wd:5
[ 11.882852] disk 0, o:1, dev:sdc2
[ 11.882854] disk 1, o:1, dev:sdd2
[ 11.882855] disk 2, o:1, dev:sde2
[ 11.882856] disk 3, o:1, dev:sdf2
[ 11.882858] disk 4, o:1, dev:sdg2
[ 11.882859] disk 5, o:1, dev:sdb2
[ 11.882860] ...ok start reshape thread
[ 11.882905] md2: detected capacity change from 0 to 34376515584
[ 11.882970] md: md2 switched to read-write mode.
[ 11.883452] md: reshape of RAID array md2
[ 11.883453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 11.883455] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[ 11.883459] md: using 128k window, over a total of 8392704 blocks.
[ 11.954838] md2: unknown partition table
[ 12.939843] Adding 1648632k swap on /dev/sda5. Priority:-1 extents:1 across:1648632k
[ 13.142646] EXT3 FS on sda1, internal journal
[ 13.255820] loop: module loaded
[ 100.224309] e1000e 0000:00:19.0: irq 30 for MSI/MSI-X
[ 100.280126] e1000e 0000:00:19.0: irq 30 for MSI/MSI-X
[ 100.280456] ADDRCONF(NETDEV_UP): eth2: link is not ready
[ 104.940830] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[ 104.941081] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[ 143.313242] md: md_do_sync() got signal ... exiting
[ 143.373125] md: md2 stopped.
[ 143.373138] md: unbind<sdc2>
[ 143.384082] md: export_rdev(sdc2)
[ 143.384113] md: unbind<sdb2>
[ 143.400081] md: export_rdev(sdb2)
[ 143.400107] md: unbind<sdg2>
[ 143.416081] md: export_rdev(sdg2)
[ 143.416108] md: unbind<sdf2>
[ 143.432080] md: export_rdev(sdf2)
[ 143.432105] md: unbind<sde2>
[ 143.448080] md: export_rdev(sde2)
[ 143.448104] md: unbind<sdd2>
[ 143.464081] md: export_rdev(sdd2)
[ 143.464405] md2: detected capacity change from 34376515584 to 0
[ 252.687538] md: md4 stopped.
[ 252.690104] md: bind<sdd4>
[ 252.690266] md: bind<sde4>
[ 252.690415] md: bind<sdf4>
[ 252.696210] md: bind<sdg4>
[ 252.718353] md: bind<sdb4>
[ 252.723594] md: bind<sdc4>
[ 332.729180] md: md4 stopped.
[ 332.729190] md: unbind<sdc4>
[ 332.740090] md: export_rdev(sdc4)
[ 332.740165] md: unbind<sdb4>
[ 332.752030] md: export_rdev(sdb4)
[ 332.752092] md: unbind<sdg4>
[ 332.768081] md: export_rdev(sdg4)
[ 332.768140] md: unbind<sdf4>
[ 332.784081] md: export_rdev(sdf4)
[ 332.784139] md: unbind<sde4>
[ 332.800081] md: export_rdev(sde4)
[ 332.800141] md: unbind<sdd4>
[ 332.816089] md: export_rdev(sdd4)
[ 556.983627] md: md4 stopped.
[ 556.988921] md: bind<sdd4>
[ 556.989094] md: bind<sde4>
[ 556.989239] md: bind<sdf4>
[ 556.989391] md: bind<sdg4>
[ 556.989642] md: bind<sdb4>
[ 556.989787] md: bind<sdc4>
# mdadm -E /dev/sd[cd]4
/dev/sdc4:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : ec600d5d:00cc3fc7:862a4878:9d191184
Name : citrouille:4
Creation Time : Wed Sep 15 17:28:55 2010
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 470429347 (224.32 GiB 240.86 GB)
Array Size : 1881714688 (897.27 GiB 963.44 GB)
Used Dev Size : 470428672 (224.32 GiB 240.86 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : e8ef9525:cfb44c96:b5e209ea:b307c619
Reshape pos'n : 0
New Layout : left-symmetric
Update Time : Tue Oct 12 00:01:03 2010
Checksum : 6dfe6f7b - correct
Events : 97
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sdd4:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : ec600d5d:00cc3fc7:862a4878:9d191184
Name : citrouille:4
Creation Time : Wed Sep 15 17:28:55 2010
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 470429347 (224.32 GiB 240.86 GB)
Array Size : 1881714688 (897.27 GiB 963.44 GB)
Used Dev Size : 470428672 (224.32 GiB 240.86 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : 08a661f3:eace7b1f:26fd20a8:ac0ae049
Reshape pos'n : 0
New Layout : left-symmetric
Update Time : Tue Oct 12 00:01:03 2010
Checksum : b32a5d21 - correct
Events : 97
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAA. ('A' == active, '.' == missing)
# mdadm -V
mdadm - v3.1.4 - 31st August 2010 - with Grow_restart always 0
(hacked mdadm)
# uname -a
Linux citrouillerescue 2.6.32-5-amd64 #1 SMP Fri Sep 17 21:50:19 UTC 2010 x86_64 GNU/Linux
# cat /etc/issue.net
Debian GNU/Linux squeeze/sid
Just in case, I posted the full output of mdadm -E /dev/sd?4 here : http://pastebin.com/zV6s2Npi
Hope it helps.
--
Simon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
