On Wed, Mar 24, 2010 at 5:35 PM, Neil Brown <neilb@xxxxxxx> wrote: > > Greetings. > I find myself in the middle of two separate off-list conversations on the > same topic and it has reached the point where I think the conversations > really need to be unite and brought on-list. > > So here is my current understanding and thoughts. > > The topic is about making rebuild after a failure easier. It strikes me as > particularly relevant after the link Bill Davidsen recently forwards to the > list: > > http://blogs.techrepublic.com.com/opensource/?p=1368 > > The most significant thing I got from this was a complain in the comments > that managing md raid was too complex and hence error-prone. > > I see the issue as breaking down in to two parts. > 1/ When a device is hot plugged into the system, is md allowed to use it as > a spare for recovery? > 2/ If md has a spare device, what set of arrays can it be used in if needed. > > A typical hot plug event will need to address both of these questions in > turn before recovery actually starts. > > Part 1. > > A newly hotplugged device may have metadata for RAID (0.90, 1.x, IMSM, DDF, > other vendor metadata) or LVM or a filesystem. It might have a partition > table which could be subordinate to or super-ordinate to other metadata. > (i.e. RAID in partitions, or partitions in RAID). The metadata may or may > not be stale. It may or may not match - either strongly or weakly - > metadata on devices in currently active arrays. > > A newly hotplugged device also has a "path" which we can see > in /dev/disk/by-path. This is somehow indicative of a physical location. > This path may be the same as the path of a device which was recently > removed. It might be one of a set of paths which make up a "RAID chassis". > It might be one of a set of paths one which we happen to find other RAID > arrays. > > Some how from all of that information we need to decide if md can use the > device without asking, or possibly with a simple yes/no question, and we > need to decide what to actually do with the device. > > Options for what to do with the device include: > - write an MBR and partition table, then do something as below with > each partition > - include the device (or partition) in an array that it was previously > part of, but from which it was removed > - include the device or partition as a spare in a native-metadata array. > - add the device as a spare to a vendor-metadata array > > Part 2. > > If we have a spare device and a degraded array we need to know if it is OK > to add the device as a hot-spare to that array. > Currently this is handled (for native metadata) by 'mdadm --monitor' and > the spare-groups tag in mdadm.conf. > For vendor metadata, if the spare is already in the container then mdmon > should handle the spare assignment, but if the spare is in a different > container, 'mdadm --monitor' should move it to the right container, but > doesn't yet. > > The "spare-group" functionality works but isn't necessarily the easiest > way to express the configuration desires. People are likely to want to > specify how far a global spare can migrate using physical address: path. > > So for example you might specify a group of paths with wildcards with the > implication that all arrays which contain disks from this group of paths > are automatically in the same spare-group. > > > Configuration and State > > I think it is clear that configuration for this should go in mdadm.conf. > This would at least cover identifying groups of device by path and ways > what is allowed to be done to those devices. > It is possible that some configuration could be determined by inspecting > the hardware directly. e.g. the IMSM code currently looks for an Option > ROM show confirms that the right Intel controller is present and so the > system can boot from the IMSM device. It is possible that other > information could be gained this way so that the mdadm.conf configuration > would not need to identify paths but alternately identify some > platform-specific concept. > > The configuration would have to say what is permitted for hot-plugged > devices: nothing, re-add, claim-bare-only, claim-any-unrecognised > The configuration would also describe mobility of spares across > different device sets. > > This would add a new line type to mdadm.conf. e.g. > DOMAIN or CHASSIS or DEDICATED or something else. > The line would identify > some devices by path or platform > a metadata type that is expected here > what hotplug is allows to do > a spare-group that applies to all array which use devices from this > group/domain/chassis/thing > source for MBR? template for partitioning? or would this always > be copied from some other device in the set if hotplug= allowed > partitioning? > > State required would include > - where devices have been recently removed from, and what they were in > use for > - which arrays are currently using which device sets, though that can > be determined dynamically from inspecting active arrays. > - ?? partition tables off any devices that are in use so if they are > removed and an new device added the partition table can be > replaced. > > Usability > > The idea of being able to pull out a device and plug in a replacement and > have it all "just work" is a good one. However I don't want to be too > dependent on state that might have been saved from the old device. > I would like to also be able to point to a new device which didn't exist > before and say "use this". mdadm would use the path information to decide > which contain or set of drives was most appropriate, extract > MBR/partitioning from one of those, impose it on the new device and include > the device or partitions in the appropriate array. > > For RAID over partitions, this assumes a fairly regular configuration: all > devices partitioned the same way, and each array build out of a set of > aligned partitions (e.g. /dev/sd[bcde]2 ). > One of the strength of md is that you don't have to use such a restricted > configuration, but I think it would be very hard to reliably "to the right > thing" with an irregular set up (e.g. a raid1 over a 1T device and 2 500GB > devices in a raid0). > > So I think we should firmly limit the range of configurations for which > auto-magic stuff is done. Vendor metadata is already fairly strongly > defined. We just add a device to the vendor container and let it worry > about the detail. For native metadata we need to draw a firm line. > I think that line should be "all devices partitioned the same" but I > am open to discussion. > > If we have "mdadm --use-this-device-however" without needing to know > anything about pre-existing state, then a hot-remove would just need to > record that the device was used by arrays X and Y. Then on hot plug we could > - do nothing > - do something if metadata on device allows > - do use-this-device-however if there was a recent hot-remove of the device > - always do use-this-device-however > depending on configuration. > > Implementation > > I think we all agree that migrating spares between containers is best done > by "mdadm --monitor". It needs to be enhanced to intuit spare-group names > from "DOMAIN" declarations, and to move spares between vendor containers. > > For hot-plug, hot-unplug I prefer to use udev triggers. plug runs > mdadm --incremental /dev/whatever > which would be extended to do other clever things if allowed > Unplug would run > mdadm --force-remove /dev/whatever > which finds any arrays containing the device (or partitions?) and > fail/removes them and records the fact with a timestamp. > > However if someone has a convincing reason to build this functionality > into "mdadm --monitor" instead using libudev I am willing to listen. > > Probably the most important first step is to determine a configuration > syntax and be sure it is broad enough to cover all needs. > > I'm thinking: > DOMAIN path=glob-pattern metadata=type hotplug=mode spare-group=name > > I explicitly have "path=" in case we find there is a need to identify > devices some other way - maybe by control vendor:device or some other > content-based approach > The spare-group name is inherited by any array with devices in this > domain as long as that doesn't result it in having two different > spare-group names. > I'm not sure if "metadata=" is really needed. If all the arrays that use > these devices have the same metadata, it would be redundant to list it here. > If they use different metadata ... then what? > I guess two different DOMAIN lines could identify the same devices and > list different metadata types and given them different spare-group > names. However you cannot support hotplug of bare devices into both ... > > If it possible for multiple DOMAIN lines to identify the same device, > e.g. by having more or less specific patterns. In this case the spare-group > names are ignored if they conflict, and the hotplug mode used is the most > permissive. > > hotplug modes are: > none - ignore any hotplugged device > incr - normal incremental assembly (the default). If the device has > metadata that matches an array, try to add it to the array > replace - If above fails and a device was recently removed from this > same path, add this device to the same array(s) that the old devices > was part of > include - If the above fails and the device has not recognisable metadata > add it to any array/container that uses devices in this domain, > partitioning first if necessary. > force - as above but ignore any pre-existing metadata > > > I'm not sure that all those are needed, or are the best names. Names like > ignore, reattach, rebuild, rebuild_spare > have also been suggested. > > It might be useful to have a 'partition=type' flag to specify MBR or GPT ?? > > > There, I think that just about covers everything relevant from the various > conversations. > Please feel free to disagree or suggest new use cases or explain why this > would not work or would not be ideal. > There was a suggestion that more state needed to be stored to support > auto-rebuild (detail of each device so they can be recovered exactly after a > device is pulled and a replacement added). I'm not convinced of this but am > happy to hear more explanations. > > Thanks, > NeilBrown > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > My feeling on the entire subject matter is that this is /not/ an easy decision. Computers are rarely correct when they guess at what an administrator wants, and attempting to implement the functionality within mdadm is prone to many limitations or re-inventing the wheel. If mdadm / mdmon is part of the process at all, I think it should be used to either fork an executable (script or otherwise) which invokes the administrative actions that have been pre-determined. I believe that the default action should be to do /nothing/. That is the only safe thing to do. If an administrative framework is desired that seems to fall under a larger project goal which is likely better covered by programs more aware of the overall system state. This route also allows for a range of scalability. It may be sufficient in an initramfs context to either spawn a shell or even just wait in a recovery console after the mdadm invocation returns failure. It might also be desired to use a very simple reaction which assumes any spare of sufficient size which is added should be allocated to the largest or closest comparable area based on pre-determined preferences. At the same time, I could see the value in mapping actual physical locations to an array, remembering any missing or failed device layouts, and re-creating the same layouts on the new device. However those actions are a little above what mdadm should be operating at. With both of those viewpoints I see the following solution. The most specific action match is followed. Action-matches should be restrict-able by path wildcard, simple size comparisons, AND state for metadata. As a final deciding factor action-matches should also have an optional priority value, so that when all else matches one rule out of a set will be known to run first. The result of matching an action, once again, should be an external program or shell to allow for maximum flexibility. I am not at all opposed to adding good default choices for those actions in either binary or shell script form. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
