Through numerous and consistent customer cases, it has been noted
that on systems with above average or high load, md devices backed
with rotational media cause a significant, system-wide I/O performance
impact during resync. This includes, but is not limited to, the
installation environment when root is on a md device. For all intents
and purposes, due to drastically increased seek operations, this
behavior is completely warranted and expected. However, it does cause
resync operations to only be truly feasible on low load systems or
during downtime. As devices grow larger, resyncs are taking longer,
requiring feasibility to extend into production uptime. So, taking
this into account, for rotational devices, 200000 is no longer a
reasonable limit. It's been found that when these performance issues
are encountered, in virtually all cases, the issue can be completely
resolved by setting a sync_speed_max value somewhere in between 50000
and 100000, the lower it's set, the better the performance gets, as
expected.
Avoid these performance hits by persistently setting rotational devices,
via a udev rule, to a more reasonable value of 100000. The rule will
check if the rotational sysfs value equals 1, then check if a
local value has already been set. This local check should afford us some
form of backward compatibility, preventing an override of already set,
per md device values. If both these criteria are matched, it will echo
the desired value into sysfs. One note is that this rule will override
the system-wide sysctl values, so if it's to be overridden,
the end user will have to create a new rule in /etc/udev/rules.d/ to
override or echo a new value in manually.
Going forward, this change does two things of note. Lowering the value in
this way should drastically decrease the number of rotational media end users
that hit a performance impact due to resync. It will also allow for a future
seamless increase in the kernel default for sync_speed_max. This will surely
be needed sooner rather than later as SSD/NVM devices become more and more common.
Signed-off-by: John Pittman <jpittman@xxxxxxxxxx>
---
udev-md-raid-arrays.rules | 3 +++
1 file changed, 3 insertions(+)
diff --git a/udev-md-raid-arrays.rules b/udev-md-raid-arrays.rules
index c95ec7b..1295709 100644
--- a/udev-md-raid-arrays.rules
+++ b/udev-md-raid-arrays.rules
@@ -32,6 +32,9 @@ OPTIONS+="watch"
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
+# Set persistent sync_speed_max for rotational media
+ATTR{queue/rotational}=="1", ATTR{md/sync_speed_max}!="*local*", RUN+="/bin/sh -c '/bin/echo 100000 > /sys%p/md/sync_speed_max'"
+
ENV{MD_LEVEL}=="raid[1-9]*", ENV{SYSTEMD_WANTS}+="mdmonitor.service"
# Tell systemd to run mdmon for our container, if we need it.
--
2.7.5
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
