Re: Raid 5 to 6 migration

[Dominique <dcouot@xxxxxxxxxxx>]

On 04/02/2013 19:39, Phil Turmel wrote:
> On 02/04/2013 07:51 AM, Dominique wrote:
> > Back from weekend, and still with my problem. Output of lsdrv is as follow:
> Yuck, mangled utf-8.  Does it look that way on your console?
> [output w/ fixed tree characters pasted below]
>
> First item, greatest importance:  You are using Western Digital Green
> drives.  These are *unsafe* to use in raid arrays of any kind "out of
> the box".  They do *not* support SCTERC.  You *must* use a boot-up
> script to set the linux driver timeouts to two minutes or more or you
> *will* crash this array.
>
> I recommend a three minute timeout.  Something like this:
> > for x in /sys/block/sd[abcdef]/driver/timeout ; do
> > 	echo 180 > $x
> > done
> in "rc.local" or wherever your distribution likes such things.  And
> don't wait for your next reboot--execute it now.
>
> If you don't have that in place, it suggests that your array is very new
> or you are not running any regular "scrub".  It is important that you
> not be vulnerable during the reshape, as the array will be heavily
> exercised.  If you aren't regularly scrubbing, execute:
> > echo "check" >/sys/block/md2/md/sync_action
> then monitor /proc/mdstat until it completes.  (Several hours)  Then
> look at the mismatch count:
> > cat /sys/block/md2/md/mismatch_count
> It should be zero.
>
> Otherwise, your array is a simple ext4 filesystem without any apparent
> system dependencies.  So you should be able to do everything needed in
> your normal boot environment, just shutting down the services that are
> using the data in "/srv".  Once the filesystem itself is resized
> (possibly quite quickly if your array has a great deal of free space),
> the remainder of the work can occur online (with /srv mounted again and
> your services restarted).
>
> Here's your recipe:
>
> 1) Stop all services using /srv, then unmount with:
> > umount /srv
> 2) Make sure the filesystem is consistent:
> > fsck /dev/md2
> 3) Determine resizing options:
> > resize2fs -P /dev/md2
> This will report the number of blocks needed for the current contents
> (usually in 4k blocks).  I don't recommend resizing all the way to the
> minimum, as it may take much longer.  Just make sure you can shrink to
> ~10 terabytes.
>
> 4) Resize:
> > resize2fs /dev/md2 10240G
> 5) Verify Ok:
> > fsck -n /dev/md2
> 6) Instruct md to temporarily use less than the future size of md2 (but
> more than the filesystem):
> > mdadm --grow /dev/md2 --array-size=10241G
> 7) Verify Again:
> > fsck -n /dev/md2
> 8) Instruct md to reshape to raid6 and view its progress:
> > mdadm --grow /dev/md2 --level=raid6 --raid-devices=6
> > cat /proc/mdstat
>   (The reshape will continue in the background.)
>
> 9) If you need the services as soon as possible, the filesystem can be
> remounted at this point, and the services restarted.  If /srv is in your
> fstab, just use:
> > mount /srv
> 10) Depending on the speed of your components, and how heavily you use
> the array, the reshape can take several hours to days.  I would expect
> yours to take at least 7 hours, best case.  Once complete, you can
> resize once more to the maximum available (this can be done while
> mounted and the services running):
> > mdadm --grow /dev/md2 --array-size=max
> > resize2fs /dev/md2
> If you run into any problems (or questions), let us know.
>
> Regards,
>
> Phil
>
> --
>
> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 6 Series/C200
> Series Chipset Family SATA AHCI Controller (rev 05)
> ├scsi 0:0:0:0 ATA      WDC WD30EZRX-00M {WD-WCAWZ1618149}
> │└sda 2.73t [8:0] Partitioned (gpt)
> │ ├sda1 95.37m [8:1] vfat {89EF-00F4}
> │ │└Mounted as /dev/sda1 @ /boot/efi
> │ ├sda2 29.80g [8:2] MD raid1 (0/6) (w/ sdd2,sde2,sdc2,sdb2,sdf2)
> in_sync 'solipym:0' {d2e6885b-2d25-6c5a-5f3d-e9a9a5daa736}
> │ │└md0 29.80g [9:0] MD v1.2 raid1 (6) clean
> {d2e6885b:2d256c5a:5f3de9a9:a5daa736}
> │ │                  swap {8e565dfc-c5fa-4c76-abe9-f3a5a6e8fcff}
> │ ├sda3 186.26g [8:3] MD raid1 (0/6) (w/ sdf3,sdd3,sde3,sdc3,sdb3)
> in_sync 'solipym:1' {89d69c6f-2ea2-23f5-aec8-b67c403488ad}
> │ │└md1 186.26g [9:1] MD v1.2 raid1 (6) clean
> {89d69c6f:2ea223f5:aec8b67c:403488ad}
> │ │ │                 ext4 {ca4bff22-40d8-4b31-859e-bba063f01df1}
> │ │ ├Mounted as /dev/disk/by-uuid/ca4bff22-40d8-4b31-859e-bba063f01df1 @ /
> │ │ └Mounted as /dev/disk/by-uuid/ca4bff22-40d8-4b31-859e-bba063f01df1 @
> /var/spool/hylafax/etc
> │ └sda4 2.52t [8:4] MD raid5 (0/6) (w/ sdd4,sdb4,sde4,sdc4,sdf4) in_sync
> 'solipym:2' {84672b4c-8fae-7f38-bb4c-c911aa9d7444}
> │  └md2 12.59t [9:2] MD v1.2 raid5 (6) clean, 512k Chunk
> {84672b4c:8fae7f38:bb4cc911:aa9d7444}
> │   │                ext4 {a9cfe95c-1e62-4cf6-b2aa-65abbd0c77e4}
> │   └Mounted as /dev/md2 @ /srv
> ├scsi 1:0:0:0 ATA      WDC WD30EZRX-00M {WD-WCAWZ1382990}
> │└sdb 2.73t [8:16] Partitioned (gpt)
> │ ├sdb1 95.37m [8:17] vfat {6EFD-1659}
> │ ├sdb2 29.80g [8:18] MD raid1 (1/6) (w/ sda2,sdd2,sde2,sdc2,sdf2)
> in_sync 'solipym:0' {d2e6885b-2d25-6c5a-5f3d-e9a9a5daa736}
> │ │└md0 29.80g [9:0] MD v1.2 raid1 (6) clean
> {d2e6885b:2d256c5a:5f3de9a9:a5daa736}
> │ │                  swap {8e565dfc-c5fa-4c76-abe9-f3a5a6e8fcff}
> │ ├sdb3 186.26g [8:19] MD raid1 (1/6) (w/ sdf3,sdd3,sde3,sda3,sdc3)
> in_sync 'solipym:1' {89d69c6f-2ea2-23f5-aec8-b67c403488ad}
> │ │└md1 186.26g [9:1] MD v1.2 raid1 (6) clean
> {89d69c6f:2ea223f5:aec8b67c:403488ad}
> │ │                   ext4 {ca4bff22-40d8-4b31-859e-bba063f01df1}
> │ └sdb4 2.52t [8:20] MD raid5 (1/6) (w/ sdd4,sda4,sde4,sdc4,sdf4)
> in_sync 'solipym:2' {84672b4c-8fae-7f38-bb4c-c911aa9d7444}
> │  └md2 12.59t [9:2] MD v1.2 raid5 (6) clean, 512k Chunk
> {84672b4c:8fae7f38:bb4cc911:aa9d7444}
> │                    ext4 {a9cfe95c-1e62-4cf6-b2aa-65abbd0c77e4}
> ├scsi 2:0:0:0 ATA      WDC WD30EZRX-00M {WD-WCAWZ0995502}
> │└sdc 2.73t [8:32] Partitioned (gpt)
> │ ├sdc1 95.37m [8:33] vfat {16BF-AABE}
> │ ├sdc2 29.80g [8:34] MD raid1 (2/6) (w/ sda2,sdd2,sde2,sdb2,sdf2)
> in_sync 'solipym:0' {d2e6885b-2d25-6c5a-5f3d-e9a9a5daa736}
> │ │└md0 29.80g [9:0] MD v1.2 raid1 (6) clean
> {d2e6885b:2d256c5a:5f3de9a9:a5daa736}
> │ │                  swap {8e565dfc-c5fa-4c76-abe9-f3a5a6e8fcff}
> │ ├sdc3 186.26g [8:35] MD raid1 (2/6) (w/ sdf3,sdd3,sde3,sda3,sdb3)
> in_sync 'solipym:1' {89d69c6f-2ea2-23f5-aec8-b67c403488ad}
> │ │└md1 186.26g [9:1] MD v1.2 raid1 (6) clean
> {89d69c6f:2ea223f5:aec8b67c:403488ad}
> │ │                   ext4 {ca4bff22-40d8-4b31-859e-bba063f01df1}
> │ └sdc4 2.52t [8:36] MD raid5 (2/6) (w/ sdd4,sda4,sdb4,sde4,sdf4)
> in_sync 'solipym:2' {84672b4c-8fae-7f38-bb4c-c911aa9d7444}
> │  └md2 12.59t [9:2] MD v1.2 raid5 (6) clean, 512k Chunk
> {84672b4c:8fae7f38:bb4cc911:aa9d7444}
> │                    ext4 {a9cfe95c-1e62-4cf6-b2aa-65abbd0c77e4}
> ├scsi 3:0:0:0 ATA      WDC WD30EZRX-00M {WD-WCAWZ1118226}
> │└sdd 2.73t [8:48] Partitioned (gpt)
> │ ├sdd1 95.37m [8:49] vfat {978F-21B2}
> │ ├sdd2 29.80g [8:50] MD raid1 (3/6) (w/ sda2,sde2,sdc2,sdb2,sdf2)
> in_sync 'solipym:0' {d2e6885b-2d25-6c5a-5f3d-e9a9a5daa736}
> │ │└md0 29.80g [9:0] MD v1.2 raid1 (6) clean
> {d2e6885b:2d256c5a:5f3de9a9:a5daa736}
> │ │                  swap {8e565dfc-c5fa-4c76-abe9-f3a5a6e8fcff}
> │ ├sdd3 186.26g [8:51] MD raid1 (3/6) (w/ sdf3,sde3,sda3,sdc3,sdb3)
> in_sync 'solipym:1' {89d69c6f-2ea2-23f5-aec8-b67c403488ad}
> │ │└md1 186.26g [9:1] MD v1.2 raid1 (6) clean
> {89d69c6f:2ea223f5:aec8b67c:403488ad}
> │ │                   ext4 {ca4bff22-40d8-4b31-859e-bba063f01df1}
> │ └sdd4 2.52t [8:52] MD raid5 (3/6) (w/ sda4,sdb4,sde4,sdc4,sdf4)
> in_sync 'solipym:2' {84672b4c-8fae-7f38-bb4c-c911aa9d7444}
> │  └md2 12.59t [9:2] MD v1.2 raid5 (6) clean, 512k Chunk
> {84672b4c:8fae7f38:bb4cc911:aa9d7444}
> │                    ext4 {a9cfe95c-1e62-4cf6-b2aa-65abbd0c77e4}
> ├scsi 4:0:0:0 ATA      WDC WD30EZRX-00M {WD-WCAWZ1649385}
> │└sde 2.73t [8:64] Partitioned (gpt)
> │ ├sde1 95.37m [8:65] vfat {8875-2D50}
> │ ├sde2 29.80g [8:66] MD raid1 (4/6) (w/ sda2,sdd2,sdc2,sdb2,sdf2)
> in_sync 'solipym:0' {d2e6885b-2d25-6c5a-5f3d-e9a9a5daa736}
> │ │└md0 29.80g [9:0] MD v1.2 raid1 (6) clean
> {d2e6885b:2d256c5a:5f3de9a9:a5daa736}
> │ │                  swap {8e565dfc-c5fa-4c76-abe9-f3a5a6e8fcff}
> │ ├sde3 186.26g [8:67] MD raid1 (4/6) (w/ sdf3,sdd3,sda3,sdc3,sdb3)
> in_sync 'solipym:1' {89d69c6f-2ea2-23f5-aec8-b67c403488ad}
> │ │└md1 186.26g [9:1] MD v1.2 raid1 (6) clean
> {89d69c6f:2ea223f5:aec8b67c:403488ad}
> │ │                   ext4 {ca4bff22-40d8-4b31-859e-bba063f01df1}
> │ └sde4 2.52t [8:68] MD raid5 (4/6) (w/ sdd4,sda4,sdb4,sdc4,sdf4)
> in_sync 'solipym:2' {84672b4c-8fae-7f38-bb4c-c911aa9d7444}
> │  └md2 12.59t [9:2] MD v1.2 raid5 (6) clean, 512k Chunk
> {84672b4c:8fae7f38:bb4cc911:aa9d7444}
> │                    ext4 {a9cfe95c-1e62-4cf6-b2aa-65abbd0c77e4}
> └scsi 5:0:0:0 ATA      WDC WD30EZRX-00M {WD-WCAWZ1383151}
>   └sdf 2.73t [8:80] Partitioned (gpt)
>    ├sdf1 95.37m [8:81] vfat {A89B-DC05}
>    ├sdf2 29.80g [8:82] MD raid1 (5/6) (w/ sda2,sdd2,sde2,sdc2,sdb2)
> in_sync 'solipym:0' {d2e6885b-2d25-6c5a-5f3d-e9a9a5daa736}
>    │└md0 29.80g [9:0] MD v1.2 raid1 (6) clean
> {d2e6885b:2d256c5a:5f3de9a9:a5daa736}
>    │                  swap {8e565dfc-c5fa-4c76-abe9-f3a5a6e8fcff}
>    ├sdf3 186.26g [8:83] MD raid1 (5/6) (w/ sdd3,sde3,sda3,sdc3,sdb3)
> in_sync 'solipym:1' {89d69c6f-2ea2-23f5-aec8-b67c403488ad}
>    │└md1 186.26g [9:1] MD v1.2 raid1 (6) clean
> {89d69c6f:2ea223f5:aec8b67c:403488ad}
>    │                   ext4 {ca4bff22-40d8-4b31-859e-bba063f01df1}
>    └sdf4 2.52t [8:84] MD raid5 (5/6) (w/ sdd4,sda4,sdb4,sde4,sdc4)
> in_sync 'solipym:2' {84672b4c-8fae-7f38-bb4c-c911aa9d7444}
>     └md2 12.59t [9:2] MD v1.2 raid5 (6) clean, 512k Chunk
> {84672b4c:8fae7f38:bb4cc911:aa9d7444}
>                       ext4 {a9cfe95c-1e62-4cf6-b2aa-65abbd0c77e4}
> USB [usb-storage] Bus 002 Device 004: ID 059f:1010 LaCie, Ltd Desktop
> Hard Drive {ST3500830A            9QG6RC54}
> └scsi 6:0:0:0 ST350083 0AS
>   └sdg 465.76g [8:96] Partitioned (dos)
>    └sdg1 465.76g [8:97] ext4 {4c2f6b92-829d-4e53-b553-c07e0f571e02}
>     └Mounted as /dev/sdg1 @ /mnt
> Other Block Devices
> ├loop0 0.00k [7:0] Empty/Unknown
> ├loop1 0.00k [7:1] Empty/Unknown
> ├loop2 0.00k [7:2] Empty/Unknown
> ├loop3 0.00k [7:3] Empty/Unknown
> ├loop4 0.00k [7:4] Empty/Unknown
> ├loop5 0.00k [7:5] Empty/Unknown
> ├loop6 0.00k [7:6] Empty/Unknown
> ├loop7 0.00k [7:7] Empty/Unknown
> ├ram0 64.00m [1:0] Empty/Unknown
> ├ram1 64.00m [1:1] Empty/Unknown
> ├ram2 64.00m [1:2] Empty/Unknown
> ├ram3 64.00m [1:3] Empty/Unknown
> ├ram4 64.00m [1:4] Empty/Unknown
> ├ram5 64.00m [1:5] Empty/Unknown
> ├ram6 64.00m [1:6] Empty/Unknown
> ├ram7 64.00m [1:7] Empty/Unknown
> ├ram8 64.00m [1:8] Empty/Unknown
> ├ram9 64.00m [1:9] Empty/Unknown
> ├ram10 64.00m [1:10] Empty/Unknown
> ├ram11 64.00m [1:11] Empty/Unknown
> ├ram12 64.00m [1:12] Empty/Unknown
> ├ram13 64.00m [1:13] Empty/Unknown
> ├ram14 64.00m [1:14] Empty/Unknown
> └ram15 64.00m [1:15] Empty/Unknown
> --

Thanks for the detailled answer. I read through all of it to identify 
what I did not understood before starting. I have a few clarificacions 
for you and a few questions.

Yes, that's the way my console looks like. I usually don't have problem 
with UTF8 output (i.e. not garbled), I figured it was the result of the 
lsdrv.... Nasty to read, but it's all there.

I am running Ubuntu 12.04.1 server on this relatively new raid5 setup.
I think I understood most of the reshaping instructions, but I need to 
clarify your point related to the timeout. I tried to execute it in a 
simple bash file and got stopped at the beginning just for the lack of a 
timeoutfile?

After looking for it, I realised that sd[abcdef] were all symlinks 
pointing to another part of the system, where there is no driver 
directory. After browsing, i figured out that /sys/block/sda/device did 
led to a timeout file. Can you please confirm, it is the one we want ?

A last point I need to clarify: is the reshaping (although long) data 
destructive (backup will be done in all cases)?

Thanks,

Dominique

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html