Hi Roy, Hi Wols, thanks for your suggestions. I already tried to force the assemble, that causes also a deadlock of the mdadm process. I could manage the situation and found that it doesnt lock if I mount it readonly. Now I exported all data over the day - seems to work fine for now. Killed the array and restarted with a fresh raid10. We will see what the future brings at import time. Btw. used mdadm 4.1. Greetings Brian Am 27.02.19 um 17:59 schrieb Roy Sigurd Karlsbakk: > The event count on the different drives are almost the same - that is - the event count on the *drives* are the same, but not for md0, but then again - it's only 3 events off, so I doubt it'll make much of a problem. I'd try mdadm --assemble --force on that. Also - check if there are any more messages in dmesg/syslog regarding the drives. > > Vennlig hilsen > > roy > -- > Roy Sigurd Karlsbakk > (+47) 98013356 > http://blogg.karlsbakk.net/ > GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt > -- > Hið góða skaltu í stein höggva, hið illa í snjó rita. > > ----- Original Message ----- >> From: search@xxxxxxxxxxxxxx >> To: "Linux Raid" <linux-raid@xxxxxxxxxxxxxxx> >> Sent: Tuesday, 26 February, 2019 11:11:19 >> Subject: Raid5 reshape stuck - md0 process hang >> Hey, >> >> i am really stuck with a problem. I tried to reshape my raid5 with 3 disks, to 4 >> disks. The reshape ran this night, until 94% and got stuck in this process. I >> can see this message in syslog: >> >> „task md0_reshape blocked for more than 120 seconds“ >> >> After seeing it hanging I set the new drive to faulty and removed it during the >> hanging reshape progress - I thought the added drive was the issue. >> >> If I take a look at top, there is a md0 process which is stuck at 100% cpu. >> >> Originally the system runs on xenserver 7.2 - I booted up a newer live disk of >> ubuntu 18.04 to look how the newer mdadm version does. - Same behaviour. >> >> If I try to assemble my raid on the live dvd, there’s the same issue. Have done >> this with: mdadm —assemble —scan >> >> Here are some logs: >> >> /dev/md0: >> Version : 1.2 >> Creation Time : Tue May 31 18:18:33 2016 >> Raid Level : raid5 >> Array Size : 3906766848 (3725.78 GiB 4000.53 GB) >> Used Dev Size : 1953383424 (1862.89 GiB 2000.26 GB) >> Raid Devices : 4 >> Total Devices : 3 >> Persistence : Superblock is persistent >> >> Intent Bitmap : Internal >> >> Update Time : Tue Feb 26 09:27:51 2019 >> State : clean, degraded, reshaping >> Active Devices : 3 >> Working Devices : 3 >> Failed Devices : 0 >> Spare Devices : 0 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Consistency Policy : bitmap >> >> Reshape Status : 94% complete >> Delta Devices : 1, (3->4) >> >> Name : maulwurfshuegel:0 >> UUID : b0f13031:163a65b5:2acfee57:dc39a18f >> Events : 8194152 >> >> Number Major Minor RaidDevice State >> 0 8 33 0 active sync /dev/sdc1 >> 1 8 1 1 active sync /dev/sda1 >> 3 8 17 2 active sync /dev/sdb1 >> - 0 0 3 removed >> >> >> /dev/sda1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x45 >> Array UUID : b0f13031:163a65b5:2acfee57:dc39a18f >> Name : maulwurfshuegel:0 >> Creation Time : Tue May 31 18:18:33 2016 >> Raid Level : raid5 >> Raid Devices : 4 >> >> Avail Dev Size : 3906766991 (1862.89 GiB 2000.26 GB) >> Array Size : 5860150272 (5588.67 GiB 6000.79 GB) >> Used Dev Size : 3906766848 (1862.89 GiB 2000.26 GB) >> Data Offset : 260096 sectors >> New Offset : 257024 sectors >> Super Offset : 8 sectors >> State : clean >> Device UUID : 4f7cc9bc:78c54dbd:9cd93978:7d1ca9fb >> >> Internal Bitmap : 8 sectors from superblock >> Reshape pos'n : 5508850176 (5253.65 GiB 5641.06 GB) >> Delta Devices : 1 (3->4) >> >> Update Time : Tue Feb 26 08:46:25 2019 >> Bad Block Log : 512 entries available at offset 72 sectors >> Checksum : eed30521 - correct >> Events : 8194149 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 1 >> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing) >> >> /dev/sdb1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x4d >> Array UUID : b0f13031:163a65b5:2acfee57:dc39a18f >> Name : maulwurfshuegel:0 >> Creation Time : Tue May 31 18:18:33 2016 >> Raid Level : raid5 >> Raid Devices : 4 >> >> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB) >> Array Size : 5860150272 (5588.67 GiB 6000.79 GB) >> Used Dev Size : 3906766848 (1862.89 GiB 2000.26 GB) >> Data Offset : 260096 sectors >> New Offset : 257024 sectors >> Super Offset : 8 sectors >> State : clean >> Device UUID : 6696cd3f:a7439cef:c8d26a6e:186869ea >> >> Internal Bitmap : 8 sectors from superblock >> Reshape pos'n : 5508850176 (5253.65 GiB 5641.06 GB) >> Delta Devices : 1 (3->4) >> >> Update Time : Tue Feb 26 08:46:25 2019 >> Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present. >> Checksum : 8916e961 - correct >> Events : 8194149 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 2 >> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing) >> >> /dev/sdc1: >> Magic : a92b4efc >> Version : 1.2 >> Feature Map : 0x4d >> Array UUID : b0f13031:163a65b5:2acfee57:dc39a18f >> Name : maulwurfshuegel:0 >> Creation Time : Tue May 31 18:18:33 2016 >> Raid Level : raid5 >> Raid Devices : 4 >> >> Avail Dev Size : 3906766991 (1862.89 GiB 2000.26 GB) >> Array Size : 5860150272 (5588.67 GiB 6000.79 GB) >> Used Dev Size : 3906766848 (1862.89 GiB 2000.26 GB) >> Data Offset : 260096 sectors >> New Offset : 257024 sectors >> Super Offset : 8 sectors >> State : clean >> Device UUID : 48b8f98f:e1c5639b:29b22511:6285341e >> >> Internal Bitmap : 8 sectors from superblock >> Reshape pos'n : 5508850176 (5253.65 GiB 5641.06 GB) >> Delta Devices : 1 (3->4) >> >> Update Time : Tue Feb 26 08:46:25 2019 >> Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present. >> Checksum : 5b908bc3 - correct >> Events : 8194149 >> >> Layout : left-symmetric >> Chunk Size : 512K >> >> Device Role : Active device 0 >> Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing) >> >> What could possibly the next steps? >> >> Thanks Guys! >> >> Greetings >> Brian
