+Cc Kuai On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang <ioworker0@xxxxxxxxx> wrote: > Hi all, > > I've run into a problem with Cgroup v2 where it doesn't seem to correctly limit > I/O operations when I set both wbps and wiops for a device. However, if I only > set wbps, then everything works as expected. > > To reproduce the problem, we can follow these command-based steps: > > 1. **System Information:** > - Kernel Version and OS Release: > ``` > $ uname -r > 6.10.0-rc5+ > > $ cat /etc/os-release > PRETTY_NAME="Ubuntu 24.04 LTS" > NAME="Ubuntu" > VERSION_ID="24.04" > VERSION="24.04 LTS (Noble Numbat)" > VERSION_CODENAME=noble > ID=ubuntu > ID_LIKE=debian > HOME_URL="https://www.ubuntu.com/"; > SUPPORT_URL="https://help.ubuntu.com/"; > BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"; > PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"; > UBUNTU_CODENAME=noble > LOGO=ubuntu-logo > ``` > > 2. **Device Information and Settings:** > - List Block Devices and Scheduler: > ``` > $ lsblk > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS > sda 8:0 0 4.4T 0 disk > └─sda1 8:1 0 4.4T 0 part /data > ... > > $ cat /sys/block/sda/queue/scheduler > none [mq-deadline] kyber bfq > > $ cat /sys/block/sda/queue/rotational > 1 > ``` > > 3. **Reproducing the problem:** > - Navigate to the cgroup v2 filesystem and configure I/O settings: > ``` > $ cd /sys/fs/cgroup/ > $ stat -fc %T /sys/fs/cgroup > cgroup2fs > $ mkdir test > $ echo "8:0 wbps=10485760 wiops=100000" > io.max > ``` > In this setup: > wbps=10485760 sets the write bytes per second limit to 10 MB/s. > wiops=100000 sets the write I/O operations per second limit to 100,000. > > - Add process to the cgroup and verify: > ``` > $ echo $$ > cgroup.procs > $ cat cgroup.procs > 3826771 > 3828513 > $ ps -ef|grep 3826771 > root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash > root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef > root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color=auto 3826771 > ``` > > - Observe I/O performance using `dd` commands and `iostat`: > ``` > $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > ``` > ``` > $ iostat -d 1 -h -y -p sda > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda > 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda > 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda1 > ``` > Initially, the write speed is slow (<2MB/s) then suddenly bursts to several > hundreds of MB/s. What it would be on average? IOW how long would the whole operation in throttled cgroup take? > > - Testing with wiops set to max: > ``` > echo "8:0 wbps=10485760 wiops=max" > io.max > $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > ``` > ``` > $ iostat -d 1 -h -y -p sda > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda > 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda1 > ``` > The iostat output shows the write operations as stabilizing at around 10 MB/s, > which aligns with the defined limit of 10 MB/s. After setting wiops to max, the > I/O limits appear to work as expected. > > > Thanks, > Lance Thanks for the report Lance. Is this something you started seeing after a kernel update or switch to cgroup v2? (Or you simply noticed with this setup only?) Michal
Attachment:
signature.asc
Description: PGP signature
