Jes, Roman, thanks for responding. >> Could you provide an actual example of how this is shown? Jes, the most recent high profile example we had was a customer executing large scale kickstart installations to md devices. They found that install was taking an exceptional amount of time on these systems, putting them behind schedule. We noted fairly quickly that the issue could be worked around with a kickstart script that drops the max_sync_speed during install. We finally got them to accept the solution, but they were not happy about it. Even with the kickstart workaround, after reboot, app install crawled b/c the resync would continue at max speed. Another recent case is one where the system would crawl to the point that it was difficult/impossible to interact with the terminal; also resolved by decreasing max_sync_speed. There are a great many more that come through support that I'm not involved with. Xiao actually helped me with the install env case... it was rough. >> I also don't think this belongs in userland. It makes a lot more sense >> to me to do this in the kernel setup of defaults for the device, which >> will also allow an admin to change the sysctl setting. I actually thought the same initially but decided the opposite. :) Perfectly fine with me though to add to kernel if accepted. Detecting rotational and setting based on that should be a small change from what I saw. >>I am also curious of the impact of reverting the patch Roman Mamedov >> pointed out. I will do my best to provide results. On Fri, Mar 16, 2018 at 10:21 AM, Jes Sorensen <jes.sorensen@xxxxxxxxx> wrote: > On 03/16/2018 09:52 AM, John Pittman wrote: >> Through numerous and consistent customer cases, it has been noted >> that on systems with above average or high load, md devices backed >> with rotational media cause a significant, system-wide I/O performance >> impact during resync. This includes, but is not limited to, the >> installation environment when root is on a md device. For all intents >> and purposes, due to drastically increased seek operations, this >> behavior is completely warranted and expected. However, it does cause >> resync operations to only be truly feasible on low load systems or >> during downtime. As devices grow larger, resyncs are taking longer, >> requiring feasibility to extend into production uptime. So, taking >> this into account, for rotational devices, 200000 is no longer a >> reasonable limit. It's been found that when these performance issues >> are encountered, in virtually all cases, the issue can be completely >> resolved by setting a sync_speed_max value somewhere in between 50000 >> and 100000, the lower it's set, the better the performance gets, as >> expected. > > So I am not necessarily opposed to a change like this, however I find > the "It's been found ....." wording here rather unconvincing. Could you > provide an actual example of how this is shown? > >> Avoid these performance hits by persistently setting rotational devices, >> via a udev rule, to a more reasonable value of 100000. The rule will >> check if the rotational sysfs value equals 1, then check if a >> local value has already been set. This local check should afford us some >> form of backward compatibility, preventing an override of already set, >> per md device values. If both these criteria are matched, it will echo >> the desired value into sysfs. One note is that this rule will override >> the system-wide sysctl values, so if it's to be overridden, >> the end user will have to create a new rule in /etc/udev/rules.d/ to >> override or echo a new value in manually. > > Overriding system wide sysctl's behind the back of admins is > unacceptable and not the right way to go. If an admin sets a sysctl > value, that must be respected. > > I also don't think this belongs in userland. It makes a lot more sense > to me to do this in the kernel setup of defaults for the device, which > will also allow an admin to change the sysctl setting. > > I am also curious of the impact of reverting the patch Roman Mamedov > pointed out. > > Thanks, > Jes -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
