Hi RAIDers
I did a silly thing and now I'm stuck.
I rebooted with an unstarted/in-progress raid5 reshape from 3 disks to 4
(reshape position is 0).
The backup file is gone.
(md125 in /proc/mdstat was something like this before the reshape)
md125 : active raid5 sde7[4] sdc7[1] sdd7[2] sdf7[3](S)
1953485568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/3]
[UUU]
(md125 in /proc/mdstat was something like this after the reshape)
md125 : active raid5 sde7[4] sdc7[1] sdd7[2] sdf7[3]
1953485568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4]
[UUUU]
resync=DELAYED
Here is where I'm at now:
This assemble fails.
# OVERLAYS=/dev/mapper/sdd7 /dev/mapper/sdc7 /dev/mapper/sdf7
/dev/mapper/sde7
# mdadm --assemble /dev/md125 --force --run --verbose --invalid-backup
$OVERLAYS
mdadm: looking for devices for /dev/md125
mdadm: /dev/mapper/sdd7 is identified as a member of /dev/md125, slot 0.
mdadm: /dev/mapper/sdc7 is identified as a member of /dev/md125, slot 1.
mdadm: /dev/mapper/sdf7 is identified as a member of /dev/md125, slot 2.
mdadm: /dev/mapper/sde7 is identified as a member of /dev/md125, slot 3.
mdadm: /dev/md125 has an active reshape - checking if critical section
needs to be restored
mdadm: No backup metadata on device-3
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/mapper/sdc7 to /dev/md125 as 1
mdadm: added /dev/mapper/sdf7 to /dev/md125 as 2
mdadm: added /dev/mapper/sde7 to /dev/md125 as 3
mdadm: added /dev/mapper/sdd7 to /dev/md125 as 0
mdadm: failed to RUN_ARRAY /dev/md125: Invalid argument
This assemble runs, but it must be wrong, as I end up with no
resync/reshape.
# mdadm --assemble /dev/md125 --force --run --verbose --invalid-backup
--update=revert-reshape $OVERLAYS
(md125 in /proc/mdstat after revert-reshape)
md125 : active raid5 dm-9[4] dm-7[1] dm-6[2] dm-8[3]
2930228352 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4]
[UUUU]
My disks are happy, so no smartctl data attached.
Attached:
- /proc/mdstat output
- mdadm --examine output of sd[cdef]7 (partitions for md125)
- mdadm --examine output of sd[cdef]5 (partitions for md127)
- mdadm --detail output of /dev/md125 (the array in question)
- mdadm --detail output of /dev/md125 (after revert-reshape)
- mdadm --detail output of /dev/md127 (a similar but working array)
Any insights welcome.
Cheers
- Angus
/dev/md124:
Version : 1.2
Creation Time : Thu Sep 28 16:36:53 2017
Raid Level : raid1
Array Size : 976646528 (931.40 GiB 1000.09 GB)
Used Dev Size : 976646528 (931.40 GiB 1000.09 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sat Dec 14 07:34:30 2019
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Consistency Policy : resync
Name : bod4.lan:5
UUID : 328c13f2:bafb619f:1f290402:30ebe5e7
Events : 118229
Number Major Minor RaidDevice State
2 8 40 0 active sync /dev/sdc8
1 8 56 1 active sync /dev/sdd8
/dev/md125:
Version : 1.2
Raid Level : raid0
Total Devices : 4
Persistence : Superblock is persistent
State : inactive
Working Devices : 4
Delta Devices : 1, (-1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 64K
Name : bod4:3
UUID : 5878ce72:2adf1c62:3cdd079a:e8d2878b
Events : 158195
Number Major Minor RaidDevice
- 253 7 - /dev/dm-7
- 253 8 - /dev/dm-8
- 253 6 - /dev/dm-6
- 253 9 - /dev/dm-9
/dev/md125:
Version : 1.2
Creation Time : Sat Jul 25 16:57:21 2015
Raid Level : raid5
Array Size : 2930228352 (2794.48 GiB 3000.55 GB)
Used Dev Size : 976742784 (931.49 GiB 1000.18 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Thu Dec 12 11:42:46 2019
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Consistency Policy : resync
Name : bod4:3
UUID : 5878ce72:2adf1c62:3cdd079a:e8d2878b
Events : 158195
Number Major Minor RaidDevice State
4 253 9 0 active sync /dev/dm-9
3 253 8 1 active sync /dev/dm-8
2 253 6 2 active sync /dev/dm-6
1 253 7 3 active sync /dev/dm-7
/dev/md126:
Version : 1.2
Creation Time : Mon Feb 11 14:03:20 2013
Raid Level : raid5
Array Size : 2930228352 (2794.48 GiB 3000.55 GB)
Used Dev Size : 976742784 (931.49 GiB 1000.18 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Sat Dec 14 07:34:30 2019
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Consistency Policy : resync
Name : bod4:4
UUID : 5e89ea42:acebe8fa:86252ed7:3e4fc595
Events : 8617445
Number Major Minor RaidDevice State
4 8 86 0 active sync /dev/sdf6
3 8 54 1 active sync /dev/sdd6
5 8 38 2 active sync /dev/sdc6
2 8 70 3 active sync /dev/sde6
/dev/md127:
Version : 1.2
Creation Time : Wed Feb 6 14:34:32 2013
Raid Level : raid5
Array Size : 2916082944 (2780.99 GiB 2986.07 GB)
Used Dev Size : 972027648 (927.00 GiB 995.36 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Sat Dec 14 07:34:30 2019
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Consistency Policy : resync
Name : bod4:2
UUID : 42aabf48:8876a500:19736093:be16b779
Events : 6300504
Number Major Minor RaidDevice State
5 8 85 0 active sync /dev/sdf5
3 8 69 1 active sync /dev/sde5
4 8 53 2 active sync /dev/sdd5
6 8 37 3 active sync /dev/sdc5
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : inactive dm-7[1](S) dm-9[4](S) dm-6[2](S) dm-8[3](S)
3906971712 blocks super 1.2
md127 : active raid5 sdf5[5] sdc5[6] sdd5[4] sde5[3]
2916082944 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md124 : active (auto-read-only) raid1 sdd8[1] sdc8[2]
976646528 blocks super 1.2 [2/2] [UU]
md126 : active (auto-read-only) raid5 sde6[2] sdf6[4] sdd6[3] sdc6[5]
2930228352 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>
/dev/sdc5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 42aabf48:8876a500:19736093:be16b779
Name : bod4:2
Creation Time : Wed Feb 6 14:34:32 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1944055680 (927.00 GiB 995.36 GB)
Array Size : 2916082944 (2780.99 GiB 2986.07 GB)
Used Dev Size : 1944055296 (927.00 GiB 995.36 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=384 sectors
State : clean
Device UUID : 56c7f2fb:3eb50f1d:e1008236:18af53c8
Update Time : Sat Dec 14 07:34:30 2019
Checksum : 3ea0a767 - correct
Events : 6300504
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : ARAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdc7:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 5878ce72:2adf1c62:3cdd079a:e8d2878b
Name : bod4:3
Creation Time : Sat Jul 25 16:57:21 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953485856 (931.49 GiB 1000.18 GB)
Array Size : 2930228352 (2794.48 GiB 3000.55 GB)
Used Dev Size : 1953485568 (931.49 GiB 1000.18 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=288 sectors
State : clean
Device UUID : 7c01f04c:999439f3:a9335d69:b5c4f4ff
Reshape pos'n : 0
Delta Devices : 1 (3->4)
Update Time : Thu Dec 12 11:42:46 2019
Checksum : 1e1163c6 - correct
Events : 158195
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 42aabf48:8876a500:19736093:be16b779
Name : bod4:2
Creation Time : Wed Feb 6 14:34:32 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1944055680 (927.00 GiB 995.36 GB)
Array Size : 2916082944 (2780.99 GiB 2986.07 GB)
Used Dev Size : 1944055296 (927.00 GiB 995.36 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=384 sectors
State : clean
Device UUID : c03f487c:4042ebd0:3c3a5f8d:4201ee3a
Update Time : Sat Dec 14 07:34:30 2019
Checksum : 3c493856 - correct
Events : 6300504
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : ARAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdd7:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 5878ce72:2adf1c62:3cdd079a:e8d2878b
Name : bod4:3
Creation Time : Sat Jul 25 16:57:21 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953485856 (931.49 GiB 1000.18 GB)
Array Size : 2930228352 (2794.48 GiB 3000.55 GB)
Used Dev Size : 1953485568 (931.49 GiB 1000.18 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=288 sectors
State : clean
Device UUID : 8aa2e77d:ba87675c:62dce495:2961e894
Reshape pos'n : 0
Delta Devices : 1 (3->4)
Update Time : Thu Dec 12 11:42:46 2019
Checksum : 79b23d22 - correct
Events : 158195
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 42aabf48:8876a500:19736093:be16b779
Name : bod4:2
Creation Time : Wed Feb 6 14:34:32 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1944055680 (927.00 GiB 995.36 GB)
Array Size : 2916082944 (2780.99 GiB 2986.07 GB)
Used Dev Size : 1944055296 (927.00 GiB 995.36 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=384 sectors
State : clean
Device UUID : 6d0b70f0:60e04c60:9658c425:cb8801c6
Update Time : Sat Dec 14 07:34:30 2019
Checksum : 63494806 - correct
Events : 6300504
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sde7:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 5878ce72:2adf1c62:3cdd079a:e8d2878b
Name : bod4:3
Creation Time : Sat Jul 25 16:57:21 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953485856 (931.49 GiB 1000.18 GB)
Array Size : 2930228352 (2794.48 GiB 3000.55 GB)
Used Dev Size : 1953485568 (931.49 GiB 1000.18 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=288 sectors
State : clean
Device UUID : dc488de5:35647e20:0a38e91e:285b0d43
Reshape pos'n : 0
Delta Devices : 1 (3->4)
Update Time : Thu Dec 12 11:42:46 2019
Checksum : dc981592 - correct
Events : 158195
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf5:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 42aabf48:8876a500:19736093:be16b779
Name : bod4:2
Creation Time : Wed Feb 6 14:34:32 2013
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1944055680 (927.00 GiB 995.36 GB)
Array Size : 2916082944 (2780.99 GiB 2986.07 GB)
Used Dev Size : 1944055296 (927.00 GiB 995.36 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=384 sectors
State : clean
Device UUID : 080f2c50:46569fce:fcb53c5b:7a10d2cd
Update Time : Sat Dec 14 07:34:30 2019
Checksum : 6ea2aa3d - correct
Events : 6300504
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : ARAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdf7:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 5878ce72:2adf1c62:3cdd079a:e8d2878b
Name : bod4:3
Creation Time : Sat Jul 25 16:57:21 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953485856 (931.49 GiB 1000.18 GB)
Array Size : 2930228352 (2794.48 GiB 3000.55 GB)
Used Dev Size : 1953485568 (931.49 GiB 1000.18 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=288 sectors
State : clean
Device UUID : e8110ad3:be65238e:22944f3c:eb788c62
Reshape pos'n : 0
Delta Devices : 1 (3->4)
Update Time : Thu Dec 12 11:42:46 2019
Checksum : 749f5a04 - correct
Events : 158195
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
