Document the filesystem I/O controller: description, usage, design, etc. Signed-off-by: Andrea Righi <righi.andrea@xxxxxxxxx> --- Documentation/cgroup-v1/fsio-throttle.txt | 142 ++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 Documentation/cgroup-v1/fsio-throttle.txt diff --git a/Documentation/cgroup-v1/fsio-throttle.txt b/Documentation/cgroup-v1/fsio-throttle.txt new file mode 100644 index 000000000000..4f33cae2adea --- /dev/null +++ b/Documentation/cgroup-v1/fsio-throttle.txt @@ -0,0 +1,142 @@ + + Filesystem I/O throttling controller + +---------------------------------------------------------------------- +1. OVERVIEW + +This controller allows to limit filesystem I/O of mounted devices of specific +process containers (cgroups [1]) enforcing delays to the processes that exceed +the limits defined for their cgroup. + +The goal of the filesystem I/O controller is to improve performance +predictability from applications' point of view and provide performance +isolation of different control groups sharing the same filesystems. + +---------------------------------------------------------------------- +2. DESIGN + +I/O activity generated by READs is evaluated at the block layer, WRITEs are +evaluated when a page changes from clear to dirty (rewriting a page that was +already dirty doesn't generate extra I/O activity). + +Throttling is always performed at the VFS layer. + +This solution has the advantage of always being able to determine the +task/cgroup that originally generated the I/O request and it prevents +filesystem locking contention and potential priority inversion problems +(example: journal I/O being throttled that may slow down the entire system). + +The downside of this solution is that the controller is more fuzzy (compared to +the blkio controller) and it allows I/O bursts that may happen at the I/O +scheduler layer. + +---------------------------------------------------------------------- +2.1. TOKEN BUCKET THROTTLING + + Tokens (I/O rate) - <mb_per_second> + o + o + o + ....... <--. + \ / | Bucket size (burst limit) + \ooo/ | <bucket_size_in_mb> + --- <--' + |ooo + Incoming --->|---> Conforming + I/O |oo I/O + requests -->|--> requests + | + ---->| + +Token bucket [2] throttling: <mb_per_second> tokens are added to the bucket +every seconds; the bucket can hold at the most <bucket_size_in_mb> tokens; I/O +requests are accepted if there are available tokens in the bucket; when a +request of N bytes arrives, N tokens are removed from the bucket; if less than +N tokens are available in the bucket, the request is delayed until a sufficient +amount of token is available again in the bucket. + +---------------------------------------------------------------------- +3. USER INTERFACE + +A new I/O limit (in MB/s) can be defined using the file: +- fsio.max_mbs + +The syntax of a throttling policy is the following: + +"<major>:<minor> <mb_per_second> <bucket_size_in_mb>" + +Examples: + +- set a maximum I/O rate of 10MB/s on /dev/sda (8:0), bucket size = 10MB: + + # echo "8:0 10 10" > /sys/fs/cgroup/cg1/fsio.max_mbs + +- remove the I/O limit defined for /dev/sda (8:0): + + # echo "8:0 0 0" > /sys/fs/cgroup/cg1/fsio.max_mbs + +---------------------------------------------------------------------- +4. Additional parameters + +---------------------------------------------------------------------- +4.1. Sleep timeslice + +Sleep timeslice is a configurable parameter that allows to decide the minimum +time of sleep to enforce to throttled tasks. Tasks will never be put to sleep +for less than the sleep timeslice. Moreover wakeup timers will be always +aligned to a multiple of the sleep timeslice. + +Increasing the sleep timeslice has the advantage of reducing the overhead of +the controller: with a more coarse-grained control, less timers are created to +wake-up tasks, that means less softirq pressure in the system and less overhead +introduced. However, a bigger sleep timeslice makes the controller more fuzzy +since throttled tasks are going to receive less throttling events with larger +sleeps. + +The parameter can be changed via: +/sys/module/fsio_throttle/parameters/throttle_timeslice_ms + +The default value is 250 ms. + +Example: + - set the sleep timeslice to 1s: + + # echo 1000 > /sys/module/fsio_throttle/parameters/throttle_timeslice_ms + +---------------------------------------------------------------------- +4.2. Sleep timeframe + +This parameter defines maximum time to sleep for a throttled task. + +The parameter can be changed via: +/sys/module/fsio_throttle/parameters/throttle_timeslice_ms + +The default value is 2 sec. + +Example: + - set the sleep timeframe to 5s: + + # echo 5000 > /sys/module/fsio_throttle/parameters/throttle_timeframe_ms + +4.3. Throttle kernel threads + +By default kernel threads are never throttled or accounted for any I/O +activity. It is possible to change this behavior by setting 1 to: + +/sys/module/fsio_throttle/parameters/throttle_kernel_threads + +It is strongly recommended to not change this setting unless you know what you +are doing. + +---------------------------------------------------------------------- +5. TODO + +- Integration with the blkio controller +- Provide distinct read/write limits, as well as MBs vs iops +- Provide additional statistics in cgroupfs + +---------------------------------------------------------------------- +6. REFERENCES + +[1] Documentation/admin-guide/cgroup-v2.rst +[2] http://en.wikipedia.org/wiki/Token_bucket -- 2.17.1
