On Mon, Apr 10, 2017 at 11:04:30AM +1000, NeilBrown wrote: >On Sat, Apr 08 2017, LM wrote: > >> Hi, >> >> I have a 4 disk RAID5, the used dev size is 640.05 GB. Now I want to >> replace the 4 disks by 4 disks with a size of 2TB each. >> >> As far as I understand the man page, this can be achieved by replacing >> the devices one after another and for each device rebuild the degraded >> array with: >> >> mdadm /dev/md0 --add /dev/sdX1 >> >> Then the level change can be done together with growing the array: >> >> mdadm --grow /dev/md0 --level=raid6 --backup-file=/root/backup-md0 >> >> Does this work? >> >> I am asking if it works, because the man page also says: >> >>> mdadm --grow /dev/md4 --level=6 --backup-file=/root/backup-md4 >>> The array /dev/md4 which is currently a RAID5 array will >>> be converted to RAID6. There should normally already be >>> a spare drive attached to the array as a RAID6 needs one >>> more drive than a matching RAID5. >> >> And in my case only the size of disks is increased, not their number. >> > >Yes, it probably works, and you probably don't need a backup file. >Though you might need to explicitly tell mdadm to keep the number of >devices unchanged by specifying "--raid-disk=4". > >You probably aren't very encouraged that I say "probably" and "might", >and this is deliberate. > >I recommend that you crate 4 10Meg files, use losetup to create 10M >devices, and build a RAID5 over them with --size=5M. >Then try the --grow --level=6 command, and see what happens. >If you mess up, you can easily start from scratch and try again. >If it works, you can have some confidence that the same process will >have the same result on real devices. > >NeilBrown Thx, I tried what you suggested and found out it works like this: * Grow the RAID5 to its maximum size. (mdadm will add a spare device which it later will refuse to remove if the array size is not reduced): * Level change RAID5 -> RAID6 (will create a degraded 5 disk array, despite --raid-disk=4) * Reduce the array size so the 5th disk can be removed * Remove the 5th disk and normalize the layout Here is the full log: Create 4x 10M files: # fallocate -l 10M A # fallocate -l 10M B # fallocate -l 10M C # fallocate -l 10M D Create 4x 10M devices: # losetup /dev/loop10 A # losetup /dev/loop11 B # losetup /dev/loop12 C # losetup /dev/loop13 D Create a 4 disk RAID5 using 5M of each device: # mdadm --create /dev/md/test --level=raid5 --size=5M --raid-devices=4 /dev/loop10 /dev/loop11 /dev/loop12 /dev/loop13 > mdadm: largest drive (/dev/loop10) exceeds size (5120K) by more than 1% > Continue creating array? y > mdadm: Defaulting to version 1.2 metadata > mdadm: array /dev/md/test started. Create a FS on the RAID: # mkfs.ext4 -T small /dev/md/test > mke2fs 1.43.3 (04-Sep-2016) > Creating filesystem with 15360 1k blocks and 3840 inodes > Filesystem UUID: 0d538322-2e07-463d-9f56-b9d22f5c9f8f > Superblock backups stored on blocks: > 8193 > > Allocating group tables: done > Writing inode tables: done > Creating journal (1024 blocks): done > Writing superblocks and filesystem accounting information: done Mount the RAID: # mount /dev/md/test test/ # ls -al test/ > total 13 > drwxr-xr-x 3 root root 1024 10. Apr 22:50 . > drwxrwxrwt 5 root root 240 10. Apr 22:49 .. > drwx------ 2 root root 12288 10. Apr 22:50 lost+found Store some file on the RAID to see if it survives: # cd test/ # wget https://www.kernel.org/theme/images/logos/tux.png > --2017-04-10 22:53:18-- https://www.kernel.org/theme/images/logos/tux.png > Resolving www.kernel.org (www.kernel.org)... 147.75.205.195, 2604:1380:2000:f000::7 > Connecting to www.kernel.org (www.kernel.org)|147.75.205.195|:443... connected. > HTTP request sent, awaiting response... 200 OK > Length: 8657 (8,5K) [image/png] > Saving to: ‘tux.png’ > > tux.png 100%[================================================>] 8,45K --.-KB/s in 0,001s > > 2017-04-10 22:53:19 (6,21 MB/s) - ‘tux.png’ saved [8657/8657] # feh test/tux.png # cd .. # umount test Details about the RAID: # mdadm --detail /dev/md/test > /dev/md/test: > Version : 1.2 > Creation Time : Mon Apr 10 22:50:39 2017 > Raid Level : raid5 > Array Size : 15360 (15.00 MiB 15.73 MB) > Used Dev Size : 5120 (5.00 MiB 5.24 MB) > Raid Devices : 4 > Total Devices : 4 > Persistence : Superblock is persistent > > Update Time : Mon Apr 10 22:53:37 2017 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 512K > > Name : lars-server:test (local to host lars-server) > UUID : 49095ada:eadf4362:4f5386f5:c615e5bf > Events : 18 > > Number Major Minor RaidDevice State > 0 7 10 0 active sync /dev/loop10 > 1 7 11 1 active sync /dev/loop11 > 2 7 12 2 active sync /dev/loop12 > 4 7 13 3 active sync /dev/loop13 Grow the RAID5 to its maximum size. (mdadm will add a spare device which it later will refuse to remove if the array size is not reduced): # mdadm --grow /dev/md/test --size=7680 > mdadm: component size of /dev/md/test has been set to 7680K See if tux is still alive: # mount /dev/md/test test/ # feh test/tux.png # umount test/ Change to level 6: # mdadm --grow /dev/md/test --level=6 --raid-disk=4 --backup-file=/root/backup-md-test > mdadm: Need 1 spare to avoid degraded array, and only have 0. > Use --force to over-ride this check. Try to force it: # mdadm --grow /dev/md/test --level=6 --raid-disk=4 --backup-file=/root/backup-md-test --force > mdadm: level of /dev/md/test changed to raid6 > mdadm: this change will reduce the size of the array. > use --grow --array-size first to truncate array. > e.g. mdadm --grow /dev/md/test --array-size 15360 Reduce the array size: # mdadm --grow /dev/md/test --array-size 15360 See if tux is still alive: # mount /dev/md/test test/ # feh test/tux.png # umount test Check the size: # mdadm --detail /dev/md/test > /dev/md/test: > Version : 1.2 > Creation Time : Mon Apr 10 23:53:10 2017 > Raid Level : raid6 > Array Size : 15360 (15.00 MiB 15.73 MB) > Used Dev Size : 7680 (7.50 MiB 7.86 MB) > Raid Devices : 5 > Total Devices : 4 > Persistence : Superblock is persistent > > Update Time : Mon Apr 10 23:57:05 2017 > State : clean, degraded > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric-6 > Chunk Size : 512K > > Name : lars-server:test (local to host lars-server) > UUID : 30ce9f41:03cd27d9:0f0317a8:e4117b5c > Events : 34 > > Number Major Minor RaidDevice State > 0 7 10 0 active sync /dev/loop10 > 1 7 11 1 active sync /dev/loop11 > 2 7 12 2 active sync /dev/loop12 > 4 7 13 3 active sync /dev/loop13 > - 0 0 4 removed Now remove the 5th spare disk mdadm added: # mdadm --grow /dev/md/test --raid-disk=4 --layout=normalise --backup-file=/root/backup-md-test > mdadm: Need to backup 3072K of critical section.. See if it worked: # mdadm --detail /dev/md/test > /dev/md/test: > Version : 1.2 > Creation Time : Mon Apr 10 23:53:10 2017 > Raid Level : raid6 > Array Size : 15360 (15.00 MiB 15.73 MB) > Used Dev Size : 7680 (7.50 MiB 7.86 MB) > Raid Devices : 4 > Total Devices : 4 > Persistence : Superblock is persistent > > Update Time : Mon Apr 10 23:57:58 2017 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 512K > > Name : lars-server:test (local to host lars-server) > UUID : 30ce9f41:03cd27d9:0f0317a8:e4117b5c > Events : 46 > > Number Major Minor RaidDevice State > 0 7 10 0 active sync /dev/loop10 > 1 7 11 1 active sync /dev/loop11 > 2 7 12 2 active sync /dev/loop12 > 4 7 13 3 active sync /dev/loop13 And tux is still alive! # mount /dev/md/test test/ # feh test/tux.png # umount test And the FS is clean, too! # fsck /dev/md/test fsck from util-linux 2.28.2 e2fsck 1.43.3 (04-Sep-2016) /dev/md126: clean, 12/3840 files, 1775/15360 blocks Clean-up the test setup: # mdadm --stop /dev/md/test # losetup -d /dev/loop10 # losetup -d /dev/loop11 # losetup -d /dev/loop12 # losetup -d /dev/loop13 # rm {A..D} -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
