I all, I wrote this today and posted it at http://neil.brown.name/blog/20110216044002 I thought it might be worth posting it here too... NeilBrown ------------------------- It is about 2 years since I last published a road-map[1] for md/raid so I thought it was time for another one. Unfortunately quite a few things on the previous list remain undone, but there has been some progress. I think one of the problems with some to-do lists is that they aren't detailed enough. High-level design, low level design, implementation, and testing are all very different sorts of tasks that seem to require different styles of thinking and so are best done separately. As writing up a road-map is a high-level design task it makes sense to do the full high-level design at that point so that the tasks are detailed enough to be addressed individually with little reference to the other tasks in the list (except what is explicit in the road map). A particular need I am finding for this road map is to make explicit the required ordering and interdependence of certain tasks. Hopefully that will make it easier to address them in an appropriate order, and mean that I waste less time saying "this is too hard, I might go read some email instead". So the following is a detailed road-map for md raid for the coming months. [1] http://neil.brown.name/blog/20090129234603 Bad Block Log ------------- As devices grow in capacity, the chance of finding a bad block increases, and the time taken to recover to a spare also increases. So the practice of ejecting a device from the array as soon as a write-error is detected is getting more and more problematic. For some time we have avoided ejecting devices for read errors, by computing the expected data from elsewhere and writing back to the device - hopefully fixing the read error. However this cannot help degraded arrays and they will still eject a device (and hence fail the whole array) on a single read error. This is not good. A particular problem is that when a device does fail and we need to recover the data, we typically read all of the other blocks on all arrays. If we are going to hit any read errors, this is the most likely time, and also this is the worst possible time and it will mean that the recovery doesn't complete and the array gets stuck in a degraded state and is very susceptible to substantial loss if another failure happens. Part of the answer to this is to implement a "bad block log". This is a record of blocks that are known to be bad. i.e. either a read or a write has recently failed. Doing this allows us to just eject that block from the array rather than the whole devices. Similarly instead of failing the whole array, we can fail just one stripe. Certainly this can mean data loss, but the loss of a few K is much less traumatic than the loss of a terabyte. But using a bad block list isn't just about keeping the data loss small, it can be about keeping it to zero. If we get a write error on a block in a non-degraded array, then recording the bad block means we lose redundancy in just that stripe rather than losing it across the whole array. If we then lose a different block on a different drive, the ability to record the bad block means that we can continue without data loss. Had we needed to eject both whole drives from the array we would have lost access to all of our data. The bad block list must be recorded to stable storage to be useful, so it really needs to be on the same drives that store the data. The bad-block list for a particular device is only of any interest to that device. Keeping information about one device on another is pointless. So we don't have a bad block list for the whole array, we keep multiple lists, one for each device. It would be best to keep at least two copies of the bad block list so that if the place where the list goes bad we can keep working with the device. The same logic applies to other metadata which currently cannot be duplicated. So implementing this feature will not address metadata redundancy. A separate feature should address metadata redundancy and it can duplicate the bad block list as well as other metadata. There are doubtlessly lots of ways that the bad block list could be stored, but we need to settle on one. For externally managed metadata we need to make the list accessible via sysfs in a generic way so that a user-space program can store is as appropriate. So: for v0.90 we choose not to store a bad block list. There isn't anywhere convenient to store it and new installations of v0.90 are not really encouraged. For v1.x metadata we record in the metadata an offset (from the superblock) and a size for a table, and a 'shift' value which can be used to shift from sector addresses to block numbers. Thus the unit that is failed when an error is detected can be larger than one sector. Each entry in the table is 64bits in little-endian. The most significant 55 bits store a block number which allows for 16 exbibytes with 512byte blocks, or more if a larger shift size is used. The remaining 9 bits store a length of the bad range which can range from 1 to 512. As bad blocks can often be consecutive, this is expected to allow the list to be quite efficient. A value of all 1's cannot correctly identify a bad range of blocks and so it is used to pad out the tail of the list. The bad block list is exposed through sysfs via a directory called 'badblocks' containing several attribute files. "shift" stores the 'shift' number described above and can be set as long as the bad block list is empty. "all" and "unacknowledged" each contains a list of bad ranges, the start (in blocks, not sectors) and the length (1-512). Each can also be written to with a string of the same format as is read out. This can be used to add bad blocks to the list or to acknowledge bad blocks. Writing effectively say "this bad range is securely recorded on stable storage". All bad blocks appear in the "badblocks/all" file. Only "acknowledged" bad blocks appear in "badblocks/unacknowledged". These are ranges which appear to be bad but are not known to be stored on stable storage. When md detects a write error or a read error which it cannot correct it added the block and marks the range that it was part of as 'unacknowledged'. Any write that depends on this block is then blocked until the range is acknowledged. This ensures that an application isn't told that a write has succeeded until the data really is safe. If the bad block list is being managed by v1.x metadata internally, then the bad block list will be written out and the ranges will be acknowledged and writes unblocked automatically. If the bad block list is being managed externally, then the bad ranges will be reported in "unacknowledged_bad_blocks". The metadata handler should read this, update the on-disk metadata and write the range back to "bad_blocks". This completes the acknowledgment handshake and writes can continue. RAID1, RAID10 and RAID456 should all support bad blocks. Every read or write should perform a lookup of the bad block list. If a read finds a bad block, that device should be treated as failed for that read. This includes reads that are part of resync or recovery. If a write finds a bad block there are two possible responses. Either the block can be ignored as with reads, or we can try to write the data in the hope that it will fix the error. Always taking the second action would seem best as it allows blocks to be removed from the bad-block list, but as a failing write can take a long time, there are plenty of cases where it would not be good. To choose between these we make the simple decision that once we see a write error we never try to write to bad blocks on that device again. This may not always be the perfect strategy, but it will effectively address common scenarios. So if a bad block is marked bad due to a read error when the array was degraded, then a write (presumably from the filesystem) will have the opportunity to correct the error. However if it was marked bad due to a write error we don't risk paying the penalty of more write errors. This 'have seen a write error' status is not stored in the array metadata. So when restarting an array with some bad blocks, each device will have one chance to prove that it can correctly handle writes to a bad block. If it can, the bad block will be removed from the list and the data is that little bit safer. If it cannot, no further writes to bad blocks will be tried on the device until the next array restart. Hot Replace ----------- "Hot replace" is my name for the process of replacing one device in an array by another one without first failing the one device. Thus there can be two devices in an array filling the same 'role'. One device will contain all of the data, the other device will only contain some of it and will be undergoing a 'recovery' process. Once the second device is fully recovered it is expected that the first device will be removed from the array. This can be useful whenever you want to replace a working device with another device, without letting the array go degraded. Two obvious cases are: 1/ when you want to replace a smaller device with a larger device 2/ when you have a device with a number of bad blocks and want to replace it with a more reliable device. For '2' to be realised, the bad block log described above must be implemented, so it should be completed before this feature. Hot replace is really only needed for RAID10 and RAID456. For RAID1, simply increasing the number of devices in the array while the new device recovers, then failing the old device and decreasing the number of devices in the array is sufficient. For RAID0 or LINEAR it would be sufficient to: - stop the array - make a RAID1 without superblocks for the old and new device - re-assemble the array using the RAID1 in place of the old device. This is certainly not as convenient but is sufficient for a case that is not likely to be commonly needed. So for both the RAID10 and RAID456 modules we need: - the ability to add a device as a hot-replace device for a specific slot - the ability to record hot-spare status in the metadata. - a 'recovery' process to rebuild a device, preferably only reading from the device to be replaced, though reading from elsewhere when needed - writes to go to both primary and secondary device. - Reads to come from either if the second has recovered far enough. - to promote a secondary device to primary when the primary device (that has a hot-replace device) fails. It is not clear whether the primary should be automatically failed when the rebuild of the secondary completes. Commonly this would be ideal, but if the secondary experienced any write errors (that were recorded in the bad block log) then it would be best to leave both in place until the sysadmin resolves the situation. So in the first implementation this failing should not be automatic. The identification of a spare as a 'hot-replace' device is achieved through the 'md/dev-XXXX/slot' sysfs attribute. This is usually 'none' or a small integer identifying which slot in the array is filled by this device. A number followed by a plus (e.g. '1+') is written, then the device takes the role of a hot-spare. This syntax requires there be at most one hot spare per slot. This is a deliberate decision to manage complexity in the code. Allowing more would be of minimal value but require substantial extra complexity. v0.90 metadata is not supported. v1.x sets a 'feature bit' on the superblock of any 'hot-replace' device and naturally records in 'recover_offset' how far recovery has progressed. Externally managed metadata can support this, or not, as they choose. Reversible Reshape ------------------ It is possible to start a reshape that cannot be reversed until the reshape has completed. This is occasionally problematic. While we might hope that users would never make errors, we should try to be as forgiving as possible. Reversing a reshape that changes the number of data-devices is possible as we support both growing and shrinking and these happen in opposite directions so one is the reverse of the other. Thus at worst, such a reshape can be reversed by: - stopping the array - re-writing the metadata so it looks like the change is going in the other direction - restarting the array. However for a reshape that doesn't change the number of data devices, such as a RAID5->RAID6 conversion or a change of chunk-size, reversal is currently not possible as the change always goes in the same direction. This is currently only meaningful for RAID456, though at some later date it might be relevant for RAID10. A future change will make it possible to move the data_offset while performing a reshape, and that will sometimes require the reshape to progress in a certain direction. It is only when the data_offset is unchanged and the number of data disks is unchanged that there is any doubt about direction. In that case it needs to be explicitly stated. We need: - some way to record in the metadata the direction of the reshape - some way to ask for a reshape to be started in the reverse direction - some way to reverse a reshape that is currently happening. We have a new sysfs attribute "reshape_direction" which is "low-to-high" or "high-to-low". This defaults to "low-to-high" but will be force to "high-to-low" if the particular reshape requires it, or can be explicity set by a 'write' before the reshape commences. Once the reshape has commenced, writing a new value to this field can flip the reshape causing it to be reverted. In both v0.90 and v1.x metadata we record a reversing reshape by setting the most significant bit in reshape_position. For v0.90 we also increase the minor number to 91. For v1.x we set a feature bit as well. Change data offset during reshape --------------------------------- One of the biggest problems with reshape currently is the need for the backup file. This is a management problem as it cannot easily be found at restart, and it is a performance problem as the extra writing is expensive. In some cases we can avoid the need for a backup file completely by changing the data-offset. i.e. the location on the devices where the array data starts. For reshapes that increase the number of devices, only a single backup is required at the start. If the data_offset is moved just one chunk earlier we can do without a separate backup. This obviously requires that space was left when the array was first created. Recent versions of mdadm do leave some space with the default metadata, though more would probably be good. For reshapes that decrease the number of device, only a small backup is required right at the end of the process (at the beginning of the devices). If we move the data_offset forward by one chunk that backup too can be avoided. As we are normally reducing the size of the array in this process, we just need to reduce it a little bit more. For reshapes that neither increase of decrease the number of devices a somewhat larger change in data_offset is needed to get reasonable performance. A single chunk (of the larger chunk size) would work, but would require updating the metadata after each chunk which would be prohibitively slow unless chunks were very large. A few megabytes is probably sufficient for reasonable performance, though testing would be helpful to be sure. Current mdadm leaves no space at the start of 1.0, and about 1Meg at the start of 1.1 and 1.2 arrays. This will generally not be enough space. In these cases it will probably be best to perform the reshape in the reverse direction (helped by the previous feature). This will probably require shrinking the filesystem and the array slightly first. Future versions of mdadm should aim to leave a few metabytes free at start and end to make these reshapes work better. Moving the data offset is not possible for 0.90 metadata as it does not record a data offset. For 1.x metadata it is possible to have a different data_offset on each device. However for simplicity we will only support changing the data offset by the same amount on each device. This amount will be stored in currently-unused space in the 1.x metadata. There will be a sysfs attribute "delta_data_offset" which can be set to a number of sectors - positive or negative - to request a change in the data offset and thus avoid the need for a backup file. Bitmap of non-sync regions. --------------------------- There are a couple of reasons for having regions of an array that are known not to contain important data and are known to not necessarily be in-sync. 1/ When an array is first created it normally contains no valid data. The normal process of a 'resync' to make all parity/copies correct is largely a waste of time. 2/ When the filesystem uses a "discard" command to report that a region of the device is no-longer used it would be good to be able to pass this down to the underlying devices. To do this safely we need to record at the md level that the region is unused so we don't complain about inconsistencies and don't try to re-sync the region after a crash. If we record which regions are not in-sync in a bitmap then we can meet both of these needs. A read to a non-in-sync region would always return 0s. A 'write' to a non-in-sync region should cause that region to be resynced. Writing zeros would in some sense be ideal, but to do that we would have to block the write, which would be unfortunate. As the fs should not be reading from that area anyway, it shouldn't really matter. The granularity of the bit is probably quite hard to get right. Having it match the block size would mean that no resync would be needed and that every discard request could be handled exactly. However it could result in a very large bitmap - 30 Megabytes for a 1 terabyte device with a 4K block size. This would need to be kept in memory and looked up for every access, which could be problematic. Having a very coarse granularity would make storage and lookups more efficient. If we make sure the bitmap would fit in 4K, we would have about 32 megabytes for bit. This would mean that each time we triggered a resync it would resync for a second or two which is probably a reasonable time as it wouldn't happen very often. But it would also mean that we can only service a 'discard' request if it covers whole blocks of 32 megabytes, and I really don't know how likely that is. Actually I'm not sure if anyone knows, the jury seems to still be out on how 'discard' will work long-term. So probably aiming for a few K to a few hundred K seems reasonable. That means that the in-memory representation will have to be a two-level array. A page of pointers to other pages can cover (on a 64bit system) 512 pages or 2Meg of bitmap space which should be enough. As always we need a way to: - record the location and size of the bitmap in the metadata - allow the granularity to be set via sysfs - allow bits to be set via sysfs, and allow the current bitmap to be read via sysfs. For v0.90 metadata we won't support this as there is no room. We could possibly store about 32 bytes directly in the superblock allowing for 4Gig sections but this is unlikely to be really useful. For v1.x metadata we use 8 bytes from the 'array state info'. 4 bytes give an offset from the metadata of the start of the bitmap, 2 bytes give the space reserved for the bitmap (max 32Meg) and 2 bytes give a shift value from sectors to in-sync chunks. The actual size of the bitmap must be computed from the known size of the array and the size of the chunks. We present the bitmap in sysfs similar to the way we present the bad block list. A file 'non-sync/regions' contains start and size of regions (measured in sectors) that are known to not be in-sync. A file 'non-sync/now-in-sync' lists ranges that actually are in sync but have not been recorded in non-in-sync yet. User-space reads now-in-sync', updates the metadata, and write to 'regions'. Another file 'non-sync/to-discard' lists ranges for a which a discard request has been made. These need to be recorded in the metadata. They are then written back to the file which allows the discard request to complete. The granularity can be set via sysfs by writing to 'non-sync/chunksize'. Assume-clean when increasing array --size ----------------------------------------- When a RAID1 is created, --assume-clean can be given so that the largely-unnecessary initial resync can be avoided. When extending the size of an array with --grow --size=, there is no way to specify --assume-clean. If a non-sync bitmap (see above) is configured this doesn't matter that the extra space will simply be marked as non-in-sync. However if a non-sync bitmap is not supported by the metadata or is not configured it would be good if md/raid1 can be told not to sync the extra space - to assume that it is in-sync. So when a non-sync bitmap is not configured (the chunk-size is zero), writing to the non-sync/regions file tells md that we don't care about the region being in-sync. So the sequence: - freeze sync_action - update size - write range to non-sync/regions - unfreeze sync_action will effect a "--grow --size=bigger --assume-clean" reshape. Enable 'reshape' to also perform 'recovery'. -------------------------------------------- As a 'reshape' re-writes all the data in the array it can quite easily be used to recover to a spare device. Normally these two operations would happen separately. However if a device fails during a reshape and a spare is available it makes sense to combine them. Currently if a device fails during a reshape (leaving the array degraded but functional) the reshape will continue and complete. Then if a spare is available it will be recovered. This means a longer total time until the array is optimal. When the device fails, the reshape actually aborts, and the restarts from where it left off. If instead we allow spares to be added between the abort and the restart, and cause the 'reshape' to actually do a recovery until it reaches the point where it was already up to, then we minimise the time to getting an optimal array. When reshaping an array to fewer devices, allow 'size' to be increased -------------------------------------------------------------------- The 'size' of an array is the amount of space on each device which is used by the array. Normally the 'size' of an array cannot be set beyond the amount of space available on the smallest device. However when reshaping an array to have fewer devices it can be useful to be able to set the 'size' to be the smallest of the remaining devices - those that will still be in use after the reshape. Normally reshaping an array to have fewer devices will make the array size smaller. However if we can simultaneously increase the size of the remaining devices, the array size can stay unchanged or even grow. This can be used after replacing (ideally using hot-replace) a few devices in the array with larger devices. The net result will be a similar amount of storage using few drives, each larger than before. This should simply be a case of allowing size to be set larger when delta_disks is negative. It also requires that when converting the excess device to spares, we fail them if they are smaller than the new size. As a reshape can be reversed, we must make sure to revert the size change when reversing a reshape. Allow write-intent-bitmap to be added to an array during reshape/recovery. -------------------------------------------------------------------------- Currently it is not possible to add a write-intent-bitmap to an array that is being reshaped/resynced/recovered. There is no real justification for this, it was just easier at the time. Implementing this requires a review of all code relating to the bitmap, checking that a bitmap appearing - or disappearing - during these processes will not be a problem. As the array is quiescent when the bitmap is added, no IO will actually be happening so it *should* be safe. This should also allow a reshape to be started while a bitmap is present, as long as the reshape doesn't change the implied size of the bitmap. Support resizing of write-intent-bitmap prior to reshape -------------------------------------------------------- When we increase the 'size' of an array (the amount of the device used), that implies a change in size of the bitmap. However the kernel cannot unilaterally resize the bitmap as there may not be room. Rather, mdadm needs to be able to resize the bitmap first. This requires the sysfs interface to expose the size of the bitmap - which is currently implicit. Whether the bitmap coverage is increased by increasing the number of bits or increasing the chunk size, some updating of the bitmap storage will be necessary (particularly in the second case). So it makes sense to allow user-space to remove the bitmap then add a new bitmap with a different configuration. If there is concern about a crash between these two, writes could be suspended for the (short) duration. Currently the 'sync_size' stored in the bitmap superblock is not used. We could stop updating that, and could allow the bitmap to automatically extend up to that boundary. So: we have a well defined 'sync_size' which can be set via the superblock or via sysfs. A resize is permitted as long as there is no bitmap, or the existing bitmap has a sufficiently large sync_size. Support reshape of RAID10 arrays. --------------------------------- RAID10 arrays currently cannot be reshaped at all. It is possible to convert a 'near' mode RAID10 to RAID0, but that is about all. Some real reshape is possible and should be implemented. 1/ A 'near' or 'offset' layout can have the device size changed quite easily. 2/ Device size of 'far' arrays cannot be changed easily. Increasing device size of 'far' would require re-laying out a lot of data. We would need to record the 'old' and 'new' sizes which metadata doesn't currently allow. If we spent 8 bytes on this we could possibly manage a 'reverse reshape' style conversion here. 3/ Increasing the number of devices is much the same for all layouts. The data needs to be copied to the new location. As we currently block IO while recovery is actually happening, we could just do that for reshape as well, and make sure reshape happens in whole chunks at a time (or whatever turns out to be the minimum recordable unit). We switch to 'clean' before doing any reshape so a write will switch to 'dirty' and update the metadata. 4/ decreasing the number of devices is very much the reverse of increasing.. Here is a weird thought: We have introduced the idea that we can increase the size of remaining devices when we decrease the number of devices in the array. For 'raid10-far', the re-layout for increasing the device size is very much like that for decreasing the number of devices - just that the number doesn't actually decrease. 5/ changing layouts between 'near' and 'offset' should be manageable providing enough 'backup' space is available. We simply copy a few chunks worth of data and move reshape_position. 6/ changing layout to or from 'far' is nearly impossible... With a change in data_offset it might be possible to move one stripe at a time, always into the place just vacated. However keeping track of where we are and were it is safe to read from would be a major headache - unless it feel out with some really neat maths, which I don't think it does. So this option will be left out. So the only 'instant' conversion possible is to increase the device size for 'near' and 'offset' array. 'reshape' conversions can modify chunk size, increase/decrease number of devices and swap between 'near' and 'offset' layout providing a suitable number of chunks of backup space is available. The device-size of a 'far' layout can also be changed by a reshape providing the number of devices in not increased. Better reporting of inconsistencies. ------------------------------------ When a 'check' finds a data inconsistency it would be useful if it was reported. That would allow a sysadmin to try to understand the cause and possibly fix it. One simple approach would be to simply log all inconsistencies through the kernel logs. This would have to be limited to 'check' and possibly 'repair' passed as logging a 'sync' pass (which also find inconsistencies) can be expected to be very noisy. Another approach is to use a sysfs file to export a list of addresses. This would place some upper limit on the number of addresses that could be listed, but if there are more inconsistencies than that limit, then the details probably aren't all that important. It makes sense to follow both of these paths. - some easy-to-parse logging of inconsistencies found. - a sysfs file that lists as many inconsistencies as possible. Each inconsistency is listed as a simple sector offset. For RAID1 and RAID4/5/6, it is an offset from the start of data on the individual devices. For RAID1 and RAID10 it is an offset from the start of the array. So this can only be interpreted with a full understanding of the array layout. The actual inconsistency may be in some sector immediately following the given sector as md performs checks in blocks larger than one sector and doesn't both refining. So an process that uses this information should read forward from the address to make sure it has found all of the inconsistency. For striped array, at most 1 chunk need be examined. For non-striped (i.e. RAID1) the window size is currently 64K. The actual size can be found by dividing 'mismatch_cnt' by the number of entries in the mismatch list. This has no dependencies on other features. It relates slightly to the bad-block list as one way of dealing with an inconsistency is to tell md that a selected block in the stripe is 'bad'. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
