Hey,
i am really stuck with a problem. I tried to reshape my raid5 with 3 disks, to 4 disks. The reshape ran this night, until 94% and got stuck in this process. I can see this message in syslog:
„task md0_reshape blocked for more than 120 seconds“
After seeing it hanging I set the new drive to faulty and removed it during the hanging reshape progress - I thought the added drive was the issue.
If I take a look at top, there is a md0 process which is stuck at 100% cpu.
Originally the system runs on xenserver 7.2 - I booted up a newer live disk of ubuntu 18.04 to look how the newer mdadm version does. - Same behaviour.
If I try to assemble my raid on the live dvd, there’s the same issue. Have done this with: mdadm —assemble —scan
Here are some logs:
/dev/md0:
Version : 1.2
Creation Time : Tue May 31 18:18:33 2016
Raid Level : raid5
Array Size : 3906766848 (3725.78 GiB 4000.53 GB)
Used Dev Size : 1953383424 (1862.89 GiB 2000.26 GB)
Raid Devices : 4
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue Feb 26 09:27:51 2019
State : clean, degraded, reshaping
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : bitmap
Reshape Status : 94% complete
Delta Devices : 1, (3->4)
Name : maulwurfshuegel:0
UUID : b0f13031:163a65b5:2acfee57:dc39a18f
Events : 8194152
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 1 1 active sync /dev/sda1
3 8 17 2 active sync /dev/sdb1
- 0 0 3 removed
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : b0f13031:163a65b5:2acfee57:dc39a18f
Name : maulwurfshuegel:0
Creation Time : Tue May 31 18:18:33 2016
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906766991 (1862.89 GiB 2000.26 GB)
Array Size : 5860150272 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906766848 (1862.89 GiB 2000.26 GB)
Data Offset : 260096 sectors
New Offset : 257024 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4f7cc9bc:78c54dbd:9cd93978:7d1ca9fb
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 5508850176 (5253.65 GiB 5641.06 GB)
Delta Devices : 1 (3->4)
Update Time : Tue Feb 26 08:46:25 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : eed30521 - correct
Events : 8194149
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : b0f13031:163a65b5:2acfee57:dc39a18f
Name : maulwurfshuegel:0
Creation Time : Tue May 31 18:18:33 2016
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
Array Size : 5860150272 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906766848 (1862.89 GiB 2000.26 GB)
Data Offset : 260096 sectors
New Offset : 257024 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 6696cd3f:a7439cef:c8d26a6e:186869ea
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 5508850176 (5253.65 GiB 5641.06 GB)
Delta Devices : 1 (3->4)
Update Time : Tue Feb 26 08:46:25 2019
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 8916e961 - correct
Events : 8194149
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : b0f13031:163a65b5:2acfee57:dc39a18f
Name : maulwurfshuegel:0
Creation Time : Tue May 31 18:18:33 2016
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3906766991 (1862.89 GiB 2000.26 GB)
Array Size : 5860150272 (5588.67 GiB 6000.79 GB)
Used Dev Size : 3906766848 (1862.89 GiB 2000.26 GB)
Data Offset : 260096 sectors
New Offset : 257024 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 48b8f98f:e1c5639b:29b22511:6285341e
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 5508850176 (5253.65 GiB 5641.06 GB)
Delta Devices : 1 (3->4)
Update Time : Tue Feb 26 08:46:25 2019
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 5b908bc3 - correct
Events : 8194149
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
What could possibly the next steps?
Thanks Guys!
Greetings
Brian
