Sorry to respond to my own message but I also wanted to note that the metadata version is 1.2 and the drives have no partition table on them, the array strings were just composed of whole drives. Best, Sean On Fri, Oct 5, 2018 at 4:03 PM Sean Caron <scaron@xxxxxxxxx> wrote: > > Hi all, > > I recently had a machine drop like 14 drives out of a few different > member strings in a RAID 60 due to some spurious fault - the drives > aren't actually bad. A few of the member strings lost three, four, > five drives each and I can't get them or the top level RAID container > back up. > > In the past, when this happened, you could just locate all drives and > force the string together with a command like: > > mdadm --assemble --force /dev/mdN /dev/sdX /dev/sdY ... /dev/sdZ > > Then, once all component strings were forced back together, force back > the top level container and you'd be good to go. > > It seems like in recent kernel or md versions, now, when this happens, > the spuriously ejected drives end up stuck in State "active" instead > of State "clean" which prevents this method of recovery from working. > > I am making an assumption that the State is maybe just a single bit or > a few bits worth of flags in the RAID superblock on each physical > disk. > > What's the offset from beginning of disk for the State flag such that > I can maybe go in with a hex editor on the physical drive and toggle > it and try to force the array back together? > > I'm looking in mdadm and I don't see any mechanism in there to > forcibly toggle the drive state. Maybe this is a feature that could be > added in the future for desperate recovery purposes like this? > > Is toggling the State on individual member drives not as simple as I'm thinking? > > Of course, no backups of this LUN, so trying to do anything possible > to jam the strings back together and get this thing back online. > > Thank you! > > Sean
