Hi, I?m somewhat new to Linux and mdadm although I?ve certainly learnt a lot over the last 24 hours. I have a SuperMicro server running CentOS 7 (3.10.0-1160.11.1.e17.x86_64) with version 4.1 ? 2018-10-01 of mdadm with that was happily running with 30 8TB disk in a RAID6 configuration. (It also has boot and root on a RAID1 array ? the RAID6 array being solely for data.) It was however starting to run out of space and I investigated adding more drives to the array (it can hold a total of 45 drives). Since this device is no longer under support, obtaining the same drives as it already contained wasn?t an option and the supplier couldn?t guarantee that they could supply compatible drives. We did come to an arrangement where I would try one drive and, if it didn?t work, I could return any unopened units. I spent ages ensuring that the ones he?d suggested were as compatible as possible and I based the specs of the existing drives off the invoice for the entire system. This turned out to be a mistake as the invoice stated they were 512e drives but, as I discovered after the new drives had arrived and I was doing a final check the existing were actually 4096k drives. Of course the new drives were 512e. Bother! After a lot more reading I found out that it might be possible to reformat the new drives from 512e to 4096k using sg_format. I installed the test drive and proceeded to see if it was possible to format them to 4096k using the command sg_format ?size=4096 /dev/sd<x>. All was proceeding smoothly when my ssh session terminated due a faulty docking station killing my Ethernet connection. So I logged onto the console and restarted the sg_format which completed OK, sort of ? it did convert the disk to 4096k but it did throw an I/O error or two but they didn?t seem too concerning and I figured, if there was a problem, it would show up in the next couple of steps. I?ve since discovered the dmesg log and that indicated that there were significantly more I/O errors than I thought. Anyway, since sg_format appeared to complete OK, I moved onto the next stage which was to partition the disk with the following commands parted -a optimal /dev/sd<x> (parted) mklabel msdos (parted) mkpart primary 2048s 100% (need to check that the start is correct) (parted) align-check optimal 1 (verify alignment of partition 1) (parted) set 1 raid on (set the FLAG to RAID) (parted) print Unfortunately, I don?t have the results of the print command as my laptop unexpectedly shut down over night (it hasn?t been a good weekend) but the partitioning appeared to complete without incident. I then added the new disk to the array: mdadm --add /dev/md125 /dev/sd<x> And it completed without any problems. I then proceeded to grow the array: mdadm --grow --raid-devices=31 --backup-file=/grow_md125.bak /dev/md125 I monitored this with cat /proc/mdstat and it showed that it was reshaping but the speed was 0K/sec and the reshape didn?t progress from 0%. #cat /proc/mdstat produced: Personalities : [raid1] [raid6] [raid5] [raid4] md125 : active raid6 sdab1[30] sdw1[26] sdc1[6] sdm1[16] sdi1[12] sdz1[29] sdh1[11] sdg1[10] sds1[22] sdf1[9] sdq1[20] sdaa1[1] sdo1[18] sdu1[24] sdb1[5] sdae1[4] sdl1[15] sdj1[13] sdn1[17] sdp1[19] sdv1[25] sde1[8] sdd1[7] sdr1[21] sdt1[23] sdx1[27] sdad1[3] sdac1[2] sdy1[28] sda1[0] sdk1[14] 218789036032 blocks super 1.2 level 6, 512k chunk, algorithm 2 [31/31] [UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU] [>....................] reshape = 0.0% (1/7813894144) finish=328606806584.3min speed=0K/sec bitmap: 0/59 pages [0KB], 65536KB chunk md126 : active raid1 sdaf1[0] sdag1[1] 100554752 blocks super 1.2 [2/2] [UU] bitmap: 1/1 pages [4KB], 65536KB chunk md127 : active raid1 sdaf3[0] sdag2[1] 976832 blocks super 1.0 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: <none> # mdadm --detail /dev/md125 produced: /dev/md125: Version : 1.2 Creation Time : Wed Sep 13 15:09:40 2017 Raid Level : raid6 Array Size : 218789036032 (203.76 TiB 224.04 TB) Used Dev Size : 7813894144 (7.28 TiB 8.00 TB) Raid Devices : 31 Total Devices : 31 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Sun May 8 00:47:35 2022 State : clean, reshaping Active Devices : 31 Working Devices : 31 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Consistency Policy : bitmap Reshape Status : 0% complete Delta Devices : 1, (30->31) Name : localhost.localdomain:SW-RAID6 UUID : f9b65f55:5f257add:1140ccc0:46ca6c19 Events : 1053617 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 65 161 1 active sync /dev/sdaa1 2 65 193 2 active sync /dev/sdac1 3 65 209 3 active sync /dev/sdad1 4 65 225 4 active sync /dev/sdae1 5 8 17 5 active sync /dev/sdb1 6 8 33 6 active sync /dev/sdc1 7 8 49 7 active sync /dev/sdd1 8 8 65 8 active sync /dev/sde1 9 8 81 9 active sync /dev/sdf1 10 8 97 10 active sync /dev/sdg1 11 8 113 11 active sync /dev/sdh1 12 8 129 12 active sync /dev/sdi1 13 8 145 13 active sync /dev/sdj1 14 8 161 14 active sync /dev/sdk1 15 8 177 15 active sync /dev/sdl1 16 8 193 16 active sync /dev/sdm1 17 8 209 17 active sync /dev/sdn1 18 8 225 18 active sync /dev/sdo1 19 8 241 19 active sync /dev/sdp1 20 65 1 20 active sync /dev/sdq1 21 65 17 21 active sync /dev/sdr1 22 65 33 22 active sync /dev/sds1 23 65 49 23 active sync /dev/sdt1 24 65 65 24 active sync /dev/sdu1 25 65 81 25 active sync /dev/sdv1 26 65 97 26 active sync /dev/sdw1 27 65 113 27 active sync /dev/sdx1 28 65 129 28 active sync /dev/sdy1 29 65 145 29 active sync /dev/sdz1 30 65 177 30 active sync /dev/sdab1 NOTE: the new disk is /dev/sdab About 12 hours later, as the reshape hadn?t progressed from 0%, I looked at ways of aborting it, such as mdadm --stop /dev/md125 which didn't work so I ended up rebooting the server and this is where things really went pear-shaped. The server came up in emergency mode, which I found odd given that the boot and root should have been OK. I was able to log on as root OK but the RAID6 array ws stuck in the reshape state. I then tried mdadm --assemble --update=revert-reshape --backup-file=/grow_md125.bak --verbose --uuid= f9b65f55:5f257add:1140ccc0:46ca6c19 /dev/md125 and this produced: mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got <varying numbers> mdadm: No RAID super block on /dev/sde . . mdadm: /dev/sde1 is identified as a member of /dev/md125, slot 6 . . mdadm: /dev/md125 has an active reshape - checking if critical section needs to be restored mdadm: No backup metadata on /grow_md125.back mdadm: Failed to find backup of critical section mdadm: Failed to restore critical section for reshape, sorry. I've tried difference variations on this including mdadm --assemble --invalid-backup --force but I won't include all the different commands here because I'm having to type all this since I can't copy anything off the server while it's in Emergency Mode. I have also removed the suspect disk but this hasn't made any difference. But the closest I've come to fixing this is running mdadm /dev/md125 --assemble --invalid-backup --backup-file=/grow_md125.bak --verbose /dev/sdc1 /dev/sdd1 ....... /dev/sdaf1 and this produces: . . . mdadm: /dev/sdaf1 is identified as a member of /dev/md125, slot 4. mdadm: /dev/md125 has an active reshape - checking if critical section needs to be restored mdadm: No backup metadata on /grow_md125.back mdadm: Failed to find backup of critical section mdadm: continuing without restoring backup mdadm: added /dev/sdac1 to /dev/md125 as 1 . . . mdadm: failed to RUN_ARRAY /dev/md125: Invalid argument dmesg has this information: md: md125 stopped. md/raid:md125: reshape_position too early for auto-recovery - aborting. md: pers->run() failed ... md: md125 stopped. If you?ve stuck with me and read all this way, thank you and I hope you can help me. Regards, Bob Brand
