Hi Jani, On Sat, 11 Dec 2021 00:25:14 +0200 Jani Partanen <jiipee@xxxxxxxxxxx> wrote: > Hello, > > I recently migrated from btrfs raid-5 to md raid-5 and because I > didn't have enough drives to go directly md raid-5, I basicly did > this: > - 2 disks raid-0 > - added 3rd disk with grow when I had moved data, still raid-0 > - added 4th disk with grow when I had moved more data, still raid-0 > - added 5th disk with grow and converted to raid-5 > > Everything went fine other than I could not add internal bitmap > because array was created as raid-0 originally but thats not an > issue, I'll store bitmap to my /boot, it's raid-1 on SSD and ext4. > > Today I noticed that my array parity layout is parity-last and I > started to wonder why is that, then I did remember that there was > some strange things happening when I did add 4th drive and I made > testing and was able to replicate what happened. > > [root@nas ~]# mdadm -C /dev/md/testraid -l 0 -n 2 -N testraid > /dev/loop21 /dev/loop22 > mdadm: Defaulting to version 1.2 metadata > mdadm: array /dev/md/testraid started. > > > [root@nas ~]# mdadm -D /dev/md/testraid > /dev/md/testraid: > Version : 1.2 > Creation Time : Fri Dec 10 22:52:01 2021 > Raid Level : raid0 > Array Size : 98304 (96.00 MiB 100.66 MB) > Raid Devices : 2 > Total Devices : 2 > Persistence : Superblock is persistent > > Update Time : Fri Dec 10 22:52:01 2021 > State : clean > Active Devices : 2 > Working Devices : 2 > Failed Devices : 0 > Spare Devices : 0 > > Layout : -unknown- > Chunk Size : 512K > > Consistency Policy : none > > Name : nas:testraid (local to host nas) > UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371 > Events : 0 > > Number Major Minor RaidDevice State > 0 7 21 0 active sync /dev/loop21 > 1 7 22 1 active sync /dev/loop22 > > > [root@nas ~]# mdadm --grow /dev/md/testraid -n 3 -l 0 --add > /dev/loop23 mdadm: level of /dev/md/testraid changed to raid4 > mdadm: added /dev/loop23 > > > [root@nas ~]# mdadm -D /dev/md/testraid > /dev/md/testraid: > Version : 1.2 > Creation Time : Fri Dec 10 22:52:01 2021 > Raid Level : raid0 > Array Size : 147456 (144.00 MiB 150.99 MB) > Raid Devices : 3 > Total Devices : 3 > Persistence : Superblock is persistent > > Update Time : Fri Dec 10 22:53:33 2021 > State : clean > Active Devices : 3 > Working Devices : 3 > Failed Devices : 0 > Spare Devices : 0 > > Chunk Size : 512K > > Consistency Policy : none > > Name : nas:testraid (local to host nas) > UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371 > Events : 18 > > Number Major Minor RaidDevice State > 0 7 21 0 active sync /dev/loop21 > 1 7 22 1 active sync /dev/loop22 > 3 7 23 2 active sync /dev/loop23 > > > [root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0 --add > /dev/loop24 mdadm: level of /dev/md/testraid changed to raid4 > mdadm: added /dev/loop24 > mdadm: Need to backup 6144K of critical section.. > > > [root@nas ~]# mdadm -D /dev/md/testraid > /dev/md/testraid: > Version : 1.2 > Creation Time : Fri Dec 10 22:52:01 2021 > Raid Level : raid4 > Array Size : 196608 (192.00 MiB 201.33 MB) > Used Dev Size : 49152 (48.00 MiB 50.33 MB) > Raid Devices : 5 > Total Devices : 4 > Persistence : Superblock is persistent > > Update Time : Fri Dec 10 22:59:56 2021 > State : clean, degraded > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > > Chunk Size : 512K > > Consistency Policy : resync > > Name : nas:testraid (local to host nas) > UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371 > Events : 39 > > Number Major Minor RaidDevice State > 0 7 21 0 active sync /dev/loop21 > 1 7 22 1 active sync /dev/loop22 > 3 7 23 2 active sync /dev/loop23 > 4 7 24 3 active sync /dev/loop24 > - 0 0 4 removed > > > [root@nas ~]# mdadm --grow /dev/md/testraid -n 5 -l 5 --add > /dev/loop25 mdadm: level of /dev/md/testraid changed to raid5 > mdadm: added /dev/loop25 > > > [root@nas ~]# mdadm -D /dev/md/testraid > /dev/md/testraid: > Version : 1.2 > Creation Time : Fri Dec 10 22:52:01 2021 > Raid Level : raid5 > Array Size : 196608 (192.00 MiB 201.33 MB) > Used Dev Size : 49152 (48.00 MiB 50.33 MB) > Raid Devices : 5 > Total Devices : 5 > Persistence : Superblock is persistent > > Update Time : Fri Dec 10 23:00:39 2021 > State : clean > Active Devices : 5 > Working Devices : 5 > Failed Devices : 0 > Spare Devices : 0 > > Layout : parity-last > Chunk Size : 512K > > Consistency Policy : resync > > Name : nas:testraid (local to host nas) > UUID : b6e0c60a:da4c6fb1:9dc5ff07:adeda371 > Events : 59 > > Number Major Minor RaidDevice State > 0 7 21 0 active sync /dev/loop21 > 1 7 22 1 active sync /dev/loop22 > 3 7 23 2 active sync /dev/loop23 > 4 7 24 3 active sync /dev/loop24 > 5 7 25 4 active sync /dev/loop25 > > > Notice how array stay at raid4 degraded after I add 4th drive. Why is > this happening? It doesn't happen when I added 3rd drive, it did > reshape it through raid-4 but when it was finished it returned raid-0. When you are starting reshape of raid0 then array level is temporarily changed to level4 for simple reason, reshape algorithm is already implemented there. After migration it goes back to raid0. raid4 is degraded because we don't have parity disk. Reshape is managed by userspace. The process is working in background (fork or systemd service). After the reshape it should switch level back to raid0 but it failed in your case. Please find out how reshape is managed in all cases (fork or systemd service: mdadm-grow-continue). I think that there should be only one grow process waiting/working to reshape end. > > I also did another test where I made extra step after adding 4th disk: > > [root@nas ~]# mdadm --grow /dev/md/testraid -n 4 -l 0 > mdadm: level of /dev/md/testraid changed to raid0 > [root@nas ~]# cat /proc/mdstat > Personalities : [raid1] [raid0] [linear] [raid6] [raid5] [raid4] > md118 : active raid0 loop24[4] loop23[3] loop22[1] loop21[0] > 196608 blocks super 1.2 512k chunks > > > And when I did add 5th drive and converted array to raid-5, it > correctly defaults layout to left-symmetric. If arrays stays as degraded raid4 then it is same as raid5 with 'parity-last' consistency policy, this is why you have "parity-last" layout. It is caused by wrong level. > Is this some sort of bug or working as intented? Also why is this > "Need to backup 6144K of critical section.." happening when I add 4th > disk but not when I add 3rd disk? > It could be related with first problem, if mdadm decides to run fork, then it could produce this message. As above, please check first how reshape is managed in all cases. > distro: Fedora 35 > kernel: 5.15.6 > mdadm: v4.2-rc2 > Thanks, Mariusz
