I just realized that bursts appear to depend on the maximum values of wbps and wiops, when wiops is not `max`? Thanks, Lance On Tue, Aug 13, 2024 at 1:00 PM Lance Yang <ioworker0@xxxxxxxxx> wrote: > > Hi Kuai, > > Thanks a lot for jumping in! > > On Tue, Aug 13, 2024 at 9:37 AM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote: > > > > Hi, > > > > 在 2024/08/12 23:43, Michal Koutný 写道: > > > +Cc Kuai > > > > > > On Mon, Aug 12, 2024 at 11:00:30PM GMT, Lance Yang <ioworker0@xxxxxxxxx> wrote: > > >> Hi all, > > >> > > >> I've run into a problem with Cgroup v2 where it doesn't seem to correctly limit > > >> I/O operations when I set both wbps and wiops for a device. However, if I only > > >> set wbps, then everything works as expected. > > >> > > >> To reproduce the problem, we can follow these command-based steps: > > >> > > >> 1. **System Information:** > > >> - Kernel Version and OS Release: > > >> ``` > > >> $ uname -r > > >> 6.10.0-rc5+ > > >> > > >> $ cat /etc/os-release > > >> PRETTY_NAME="Ubuntu 24.04 LTS" > > >> NAME="Ubuntu" > > >> VERSION_ID="24.04" > > >> VERSION="24.04 LTS (Noble Numbat)" > > >> VERSION_CODENAME=noble > > >> ID=ubuntu > > >> ID_LIKE=debian > > >> HOME_URL="https://www.ubuntu.com/"; > > >> SUPPORT_URL="https://help.ubuntu.com/"; > > >> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"; > > >> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"; > > >> UBUNTU_CODENAME=noble > > >> LOGO=ubuntu-logo > > >> ``` > > >> > > >> 2. **Device Information and Settings:** > > >> - List Block Devices and Scheduler: > > >> ``` > > >> $ lsblk > > >> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS > > >> sda 8:0 0 4.4T 0 disk > > >> └─sda1 8:1 0 4.4T 0 part /data > > >> ... > > >> > > >> $ cat /sys/block/sda/queue/scheduler > > >> none [mq-deadline] kyber bfq > > >> > > >> $ cat /sys/block/sda/queue/rotational > > >> 1 > > >> ``` > > >> > > >> 3. **Reproducing the problem:** > > >> - Navigate to the cgroup v2 filesystem and configure I/O settings: > > >> ``` > > >> $ cd /sys/fs/cgroup/ > > >> $ stat -fc %T /sys/fs/cgroup > > >> cgroup2fs > > >> $ mkdir test > > >> $ echo "8:0 wbps=10485760 wiops=100000" > io.max > > >> ``` > > >> In this setup: > > >> wbps=10485760 sets the write bytes per second limit to 10 MB/s. > > >> wiops=100000 sets the write I/O operations per second limit to 100,000. > > >> > > >> - Add process to the cgroup and verify: > > >> ``` > > >> $ echo $$ > cgroup.procs > > >> $ cat cgroup.procs > > >> 3826771 > > >> 3828513 > > >> $ ps -ef|grep 3826771 > > >> root 3826771 3826768 0 22:04 pts/1 00:00:00 -bash > > >> root 3828761 3826771 0 22:06 pts/1 00:00:00 ps -ef > > >> root 3828762 3826771 0 22:06 pts/1 00:00:00 grep --color=auto 3826771 > > >> ``` > > >> > > >> - Observe I/O performance using `dd` commands and `iostat`: > > >> ``` > > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > > > > You're testing buffer IO here, and I don't see that write back cgroup is > > enabled. Is this test intentional? Why not test direct IO? > > Yes, I was testing buffered I/O and can confirm that CONFIG_CGROUP_WRITEBACK > was enabled. > > $ cat /boot/config-6.10.0-rc5+ |grep CONFIG_CGROUP_WRITEBACK > CONFIG_CGROUP_WRITEBACK=y > > We intend to configure both wbps (write bytes per second) and wiops > (write I/O operations > per second) for the containers. IIUC, this setup will effectively > restrict both their block device > I/Os and buffered I/Os. > > > Why not test direct IO? > > I was testing direct IO as well. However it did not work as expected with > `echo "8:0 wbps=10485760 wiops=100000" > io.max`. > > $ time dd if=/dev/zero of=/data/file7 bs=512M count=1 oflag=direct > 1+0 records in > 1+0 records out > 536870912 bytes (537 MB, 512 MiB) copied, 51.5962 s, 10.4 MB/s > > real 0m51.637s > user 0m0.000s > sys 0m0.313s > > $ iostat -d 1 -h -y -p sda > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 55.00 0.0k 1.8M 0.0k 0.0k 1.8M > 0.0k sda > 55.00 0.0k 1.8M 0.0k 0.0k 1.8M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 14.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 13.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 18.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 18.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 12.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M > 0.0k sda > 1804.00 0.0k 445.8M 0.0k 0.0k 445.8M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 4.00 0.0k 24.0k 0.0k 0.0k 24.0k > 0.0k sda > 4.00 0.0k 24.0k 0.0k 0.0k 24.0k > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda1 > > There are two things that confuse me. First, initially, neither the > wbps nor the wiops > reached their limits. Second, in the last second, the wbps far > exceeded the limit. > > But if I only set wbps, then everything works as expected with > `echo "8:0 wbps=10485760 wiops=max" > io.max` > > > >> ``` > > >> ``` > > >> $ iostat -d 1 -h -y -p sda > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda > > >> 7.00 0.0k 1.3M 0.0k 0.0k 1.3M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda > > >> 21.00 0.0k 1.4M 0.0k 0.0k 1.4M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda > > >> 5.00 0.0k 1.2M 0.0k 0.0k 1.2M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda > > >> 1848.00 0.0k 448.1M 0.0k 0.0k 448.1M 0.0k sda1 > > > > Looks like all dirty buffer got flushed to disk at the last second while > > the file is closed, this is expected. > > The dd command completed in less than a second, but flushing all the > dirty buffers to > disk took a much longer time. By the time the flushing was completed, > the file had > already been closed, IIUC. > > $ time dd if=/dev/zero of=/data/file5 bs=512M count=1 > 1+0 records in > 1+0 records out > 536870912 bytes (537 MB, 512 MiB) copied, 0.531944 s, 1.0 GB/s > > real 0m0.578s > user 0m0.000s > sys 0m0.576s > > $ iostat -d 1 -h -y -p sda > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 74.00 0.0k 664.0k 0.0k 0.0k 664.0k > 0.0k sda > 74.00 0.0k 664.0k 0.0k 0.0k 664.0k > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 15.00 0.0k 1.1M 0.0k 0.0k 1.1M > 0.0k sda > 15.00 0.0k 1.1M 0.0k 0.0k 1.1M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 13.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 13.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 12.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 9.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 46.00 0.0k 1.7M 0.0k 0.0k 1.7M > 0.0k sda > 46.00 0.0k 1.7M 0.0k 0.0k 1.7M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 7.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 11.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda > 5.00 0.0k 1.2M 0.0k 0.0k 1.2M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 6.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda > 11.00 0.0k 1.4M 0.0k 0.0k 1.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 8.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 15.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda > 10.00 0.0k 1.3M 0.0k 0.0k 1.3M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 49.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda > 49.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 53.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda > 53.00 0.0k 1.6M 0.0k 0.0k 1.6M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M > 0.0k sda > 1805.00 0.0k 448.4M 0.0k 0.0k 448.4M > 0.0k sda1 > > > tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn > kB_dscd Device > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda > 0.00 0.0k 0.0k 0.0k 0.0k 0.0k > 0.0k sda1 > > > >> ``` > > >> Initially, the write speed is slow (<2MB/s) then suddenly bursts to several > > >> hundreds of MB/s. > > > > > > What it would be on average? > > > IOW how long would the whole operation in throttled cgroup take? > > > > > >> > > >> - Testing with wiops set to max: > > >> ``` > > >> echo "8:0 wbps=10485760 wiops=max" > io.max > > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > > >> $ dd if=/dev/zero of=/data/file1 bs=512M count=1 & > > >> ``` > > >> ``` > > >> $ iostat -d 1 -h -y -p sda > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > > >> 48.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > > >> 40.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > > >> 41.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda > > >> 46.00 0.0k 10.0M 0.0k 0.0k 10.0M 0.0k sda1 > > >> > > >> > > >> tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd Device > > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda > > >> 55.00 0.0k 10.2M 0.0k 0.0k 10.2M 0.0k sda1 > > > > And I don't this wiops=max is the reason, what need to explain is that > > why dirty buffer got flushed to disk synchronously before the dd finish > > and close the file? > > The dd command operates in the background, and it seems that the dirty > buffers begin > to flush after the command has completed. > > > > > >> ``` > > >> The iostat output shows the write operations as stabilizing at around 10 MB/s, > > >> which aligns with the defined limit of 10 MB/s. After setting wiops to max, the > > >> I/O limits appear to work as expected. > > > > Can you give the direct IO a test? And also enable write back cgroup for > > buffer IO. > > > > Thanks, > > Kuai > > > Thanks a lot again for your time! > Lance > > > > > >> > > >> > > >> Thanks, > > >> Lance > > > > > > Thanks for the report Lance. Is this something you started seeing after > > > a kernel update or switch to cgroup v2? (Or you simply noticed with this > > > setup only?) > > > > > > > > > Michal > > > > >
