Hi,
today I had some problems with my mdadm raid5 (4disks). Firstly I try to
explaine what happened and what the result is:
One disk in my array has some bad blocks. After some hardware-changes
one of the intact disks was thrown out of the array due to a faulty
sata-cable.
I shut down the server and replaced the cable.
After booting, the removed disk wasn't re added to the array (maybe
because of different event count). --re-add doesn't work.
So I used --add.
Because of the bad blocks on one of the remaining disks, the rebuild
stops when reaching the first bad block. The re added disk is declared
as spare, 2 disks active and the disk with bad blocks as faulty.
/dev/md127:
Version : 1.2
Creation Time : Tue Apr 19 08:51:32 2011
Raid Level : raid5
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Thu Jul 16 19:02:09 2015
State : clean, FAILED
Active Devices : 2
Working Devices : 3
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Name : FiFa-Server:0
UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
Events : 107223
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 80 1 active sync /dev/sdf
5 8 32 2 active sync /dev/sdc
6 0 0 6 removed
4 8 96 - faulty /dev/sdg
6 8 64 - spare /dev/sde
In my opinion there a 3 possibilities to get the array back working. I
am not sure whether both possibilities really exist and which one is the
most promising.
- Using the 'spare'-disk as active disk. The data on the disk
should be still there.
- Ignoring the bad blocks and loose information stored in this
blocks
- force start the array without the 'spare' disk and copy the
data to backup-storage, or does the bad block will cause the
array to fail when reaching a bad block?
In the attachment you can find the output of --examine.
In can not explain why 3 disk have a Bad Block Log. According to
smart-values only sdg has Reallocated_Sector_Ct >0
Another thing I can't explain is why sdg (which is the disk with known
bad blocks) has a lower event count.
I hope I can get some great ideas how to fix my array.
Fabian
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
Name : FiFa-Server:0
Creation Time : Tue Apr 19 08:51:32 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=1200 sectors
State : clean
Device UUID : 97ccd551:8820c0e3:4ab3d67d:908a2fd9
Update Time : Thu Jul 16 19:02:09 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9d018eea - correct
Events : 107223
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x8
Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
Name : FiFa-Server:0
Creation Time : Tue Apr 19 08:51:32 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=1200 sectors
State : clean
Device UUID : 6db7566d:3c709370:12634f66:f6bfd4f6
Update Time : Thu Jul 16 19:02:09 2015
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 4b73aeab - correct
Events : 107223
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
Name : FiFa-Server:0
Creation Time : Tue Apr 19 08:51:32 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1200 sectors
State : clean
Device UUID : 27df429f:c9661838:3a8024e4:d55055e8
Update Time : Thu Jul 16 19:02:09 2015
Checksum : b4327aa4 - correct
Events : 107223
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x8
Array UUID : 839fb405:d0b1f13a:5a55ee42:fc8a2061
Name : FiFa-Server:0
Creation Time : Tue Apr 19 08:51:32 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1960 sectors, after=1200 sectors
State : clean
Device UUID : b6786dff:c6be0236:10d8f95f:63afad93
Update Time : Thu Jul 16 19:02:03 2015
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 397b22c8 - correct
Events : 107192
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
