Re-adding disk failed during reshaping causes array corruption

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Consider the following scenario:

* A new disk was added to RAID6 md array (mdadm -a /dev/md1 /dev/sdc1)
* Number of devices was increased and array started to reshape (mdadm -G /dev/md1 -n 5)
* While the array was reshaping, one of the pre-existing disks in the array failed; degraded array continued reshaping and completed the
process successfully, albeit at reduced speed (to be expected). While array was reshaping there were no signs of data corruption.
* Computer was shut down, SATA cable to the failed disk was replaced, and it came back online with only CRC errors in the SMART error log,
which indicated nothing worse than a loose connection.
* Failed disk was added to the degraded array (mdadm -a /dev/md1 /dev/sdc1)
* At this point /proc/mdstat didn't report that the array was recovering, it reported that this disk was being re-added, and the process
completed very quickly - in a minute or two.
* I am pretty sure that the previous step caused the corruption, but to be honest, I didn't check at this time - I grew filesystem to match
the array size, started copying some files and only then noticed multiple errors. 'xfs_repair -n' generated a 24M error log. All was lost.
* I tried failing the re-added disk, and the filesystem recovered [almost] completely.
* I then zeroed the superblock on this disk and added it back, normal (and lengthy) rebuild started and is currently in progress. Everything
seems to be fine.
 
Note:

1. mdadm - v4.1-rc1 - 2018-03-22, by no means the latest version, this issue may be already fixed, but as it causes severe data corruption
I'd better report it.
2. no assistance necessary, looks like I managed to fix it on my own - just reporting the bug




[Index of Archives]     [Linux RAID Wiki]     [ATA RAID]     [Linux SCSI Target Infrastructure]     [Linux Block]     [Linux IDE]     [Linux SCSI]     [Linux Hams]     [Device Mapper]     [Device Mapper Cryptographics]     [Kernel]     [Linux Admin]     [Linux Net]     [GFS]     [RPM]     [git]     [Yosemite Forum]


  Powered by Linux