Hi all,
My friend is running a Debian 8 server with 48TB raid5 array running
fine since 2016, but suddenly it failed last night around 9PM,
reporting four drives missing. See below all devices details.
I'm suspecting his disk controller with four SATA ports failed, as the
failed drives were attached on it. So, I believe these four missing
devices (sdk,sdl,sdm and sdn) are in good shape.
However they got out of sync with the array. I'm wondering how the
array got out of sync when it was not able to access these devices ?
Can we recover the array from this failure?
Thanks,
Romulo
======> Device details
/dev/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : e430cb46:00850bd2:71884e09:6e22e462
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d3d92a07 - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : ac4f56cf:b673f905:f0c6a3a9:1849e5a7
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 75a89cb0 - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : 05b22585:de8ae2ac:6729119b:4e39afda
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : f69868df - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : 7297231d:29c34e60:fddf5ffa:f76afebc
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 83a06ed7 - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : 89bf47be:d10dea17:5c617b73:ebe146b0
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 48c3d9ea - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : 5c907ee2:2244a799:25149ac9:4ea1b892
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 27485340 - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813805744 (3725.91 GiB 4000.67 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : bd71d7d9:608bd516:c259b37c:0f8cc1c1
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7df2243f - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : ad430d83:57ebd2f5:805fbad5:e5415fa9
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 46c999b4 - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : clean
Device UUID : 53cd4e42:f5683802:a7ab6de5:564fb41e
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Feb 18 00:49:45 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9778fa8f - correct
Events : 345459
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAA.A...A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813784240 (3725.90 GiB 4000.66 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : active
Device UUID : d270fc4a:dbb655be:208a0616:3870c9da
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Feb 17 20:52:56 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 4912d41a - correct
Events : 344497
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 10
Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdl:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813794480 (3725.91 GiB 4000.66 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : active
Device UUID : c9da71a4:4b3cd193:521be214:798d0e72
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Feb 17 20:52:56 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e2499f6 - correct
Events : 344497
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdm:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : active
Device UUID : deaac574:97d9b939:5bbdc0a3:06439246
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Feb 17 20:52:56 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e7c312e6 - correct
Events : 344497
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdn:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : d02e392c:6843c91f:cb0ab942:1466d37c
Name : orion:0 (local to host orion)
Creation Time : Sat Mar 12 20:26:30 2016
Raid Level : raid5
Raid Devices : 13
Avail Dev Size : 7813775024 (3725.90 GiB 4000.65 GB)
Array Size : 46882646016 (44710.78 GiB 48007.83 GB)
Used Dev Size : 7813774336 (3725.90 GiB 4000.65 GB)
Data Offset : 219136 sectors
Super Offset : 8 sectors
Unused Space : before=219048 sectors, after=43696 sectors
State : active
Device UUID : b76869a5:56675d3b:55170491:83002c92
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Feb 17 20:52:56 2019
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 52e775f9 - correct
Events : 344497
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
