On 22/07/20 08:41, Cory Derenburger wrote: > My server lost power this morning. The server is running Linux Mint > (14?) on a battery backup and I believe it shutdown before losing > power. Upon restarting the server the computer hung for a while, and > after resetting and booting up in recovery mode my RAID is now > nonfunctional. > > The server was set up years ago with a RAID 6 array built with mdadm. > To be honest I don't really know what is wrong with the array, it > seems to be an issue with disk sdc. I wanted to reach out for help to > confirm the issue and get some guidance before proceeding (or making > things worse). > > Any assistance that can help me determine what steps to take to get > this server back up and running would be greatly appreciated. It's > been 4+ since I have touched RAID, and only attempted a recovery once. > If anyone can help I would be super appreciative. https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn https://raid.wiki.kernel.org/index.php/Asking_for_help I see you've included some stuff which is helpful, but can you do everything that last page asks for. In particular, lsdrv. > > Below I'm including outputs from various commands for the 3rd disk > which seems to be the culprit > > dmesg - boot section section where first errors begin occurring > [ 2.637856] md: bind<sdd1> > [ 2.646987] random: nonblocking pool is initialized > [ 2.647432] md: bind<sde1> > [ 2.651429] md: bind<sdb1> > [ 2.863538] ata3.00: exception Emask 0x0 SAct 0x10 SErr 0x0 action 0x0 > [ 2.863594] ata3.00: irq_stat 0x40000008 > [ 2.863643] ata3.00: failed command: READ FPDMA QUEUED > [ 2.863695] ata3.00: cmd 60/08:20:08:08:00/00:00:00:00:00/40 tag 4 > ncq 4096 in > [ 2.863695] res 41/40:00:09:08:00/00:00:00:00:00/40 Emask > 0x409 (media error) <F> > [ 2.863775] ata3.00: status: { DRDY ERR } > [ 2.863822] ata3.00: error: { UNC } > [ 2.873407] ata3.00: configured for UDMA/133 > [ 2.873476] sd 2:0:0:0: [sdc] Unhandled sense code > [ 2.873525] sd 2:0:0:0: [sdc] > [ 2.873571] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > [ 2.873619] sd 2:0:0:0: [sdc] > [ 2.873665] Sense Key : Medium Error [current] [descriptor] > [ 2.873819] Descriptor sense data with sense descriptors (in hex): > [ 2.873901] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 > [ 2.874544] 00 00 08 09 > [ 2.874764] sd 2:0:0:0: [sdc] > [ 2.874811] Add. Sense: Unrecovered read error - auto reallocate failed > [ 2.874895] sd 2:0:0:0: [sdc] CDB: > [ 2.874941] Read(10): 28 00 00 00 08 08 00 00 08 00 > [ 2.875428] end_request: I/O error, dev sdc, sector 2057 > [ 2.875478] Buffer I/O error on device sdc1, logical block 1 > > cat /proc/mdstat > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] > [raid4] [raid10] > md0 : inactive sdb1[0](S) sde1[3](S) sdd1[2](S) > 5860147464 blocks super 1.2 > > {not sure why these drives are now showing as spares} This is very common when an array fails to assemble properly. Unfortunately, when there's one error, it often triggers a cascade of fake errors, and this is probably the case here. > > Below running mdstat for sdc. Checking sdb, sdd, sde appear fine. > > mdadm --examine /dev/sdc > /dev/sdc: MBR Magic : aa55 > Partition[0] : 3907027120 sectors at 2048 (type fd) > > mdadm --examine /dev/sdc1 > mdadm: No md superblock detected on /dev/sdc1. > > fdisk -l > Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes > 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0x38389fdc > > Device Boot Start End Blocks Id System > /dev/sdb1 2048 3907029167 1953513560 fd Linux raid autodetect > > Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes > 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0xd108824d > > Device Boot Start End Blocks Id System > /dev/sdc1 2048 3907029167 1953513560 fd Linux raid autodetect > > Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes > 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0x6207659a > > Device Boot Start End Blocks Id System > /dev/sdd1 2048 3907029167 1953513560 fd Linux raid autodetect > > Disk /dev/sde: 2000.4 GB, 2000398934016 bytes > 81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0xd9a4afcf > > Device Boot Start End Blocks Id System > /dev/sde1 2048 3907029167 1953513560 fd Linux raid autodetect > > > Is there other information needed to determine the issue? Where do I > go from here? > How old is linux mint? Have you kept it up-to-date? Unfortunately, it seems a lot of older systems suffer issues when the kernel is heavily patched and mdadm is not updated, and this regularly surfaces on this list where Ubuntu is concerned ... mdadm --version uname -a Make sure you have a "latest and greatest" rescue disk to hand, and we'll see what the others say. Cheers, Wol
