On Mon, Aug 29, 2022 at 10:22:37AM +0800, Yu Kuai wrote:
> From: Yu Kuai <yukuai3@xxxxxxxxxx>
>
> Test scripts:
> cd /sys/fs/cgroup/blkio/
> echo "8:0 1024" > blkio.throttle.write_bps_device
> echo $$ > cgroup.procs
> dd if=/dev/zero of=/dev/sda bs=10k count=1 oflag=direct &
> dd if=/dev/zero of=/dev/sda bs=10k count=1 oflag=direct &
>
> Test result:
> 10240 bytes (10 kB, 10 KiB) copied, 10.0134 s, 1.0 kB/s
> 10240 bytes (10 kB, 10 KiB) copied, 10.0135 s, 1.0 kB/s
>
> The problem is that the second bio is finished after 10s instead of 20s.
>
> Root cause:
> 1) second bio will be flagged:
>
> __blk_throtl_bio
> while (true) {
> ...
> if (sq->nr_queued[rw]) -> some bio is throttled already
> break
> };
> bio_set_flag(bio, BIO_THROTTLED); -> flag the bio
>
> 2) flagged bio will be dispatched without waiting:
>
> throtl_dispatch_tg
> tg_may_dispatch
> tg_with_in_bps_limit
> if (bps_limit == U64_MAX || bio_flagged(bio, BIO_THROTTLED))
> *wait = 0; -> wait time is zero
> return true;
>
> commit 9f5ede3c01f9 ("block: throttle split bio in case of iops limit")
> support to count split bios for iops limit, thus it adds flagged bio
> checking in tg_with_in_bps_limit() so that split bios will only count
> once for bps limit, however, it introduce a new problem that io throttle
> won't work if multiple bios are throttled.
>
> In order to fix the problem, handle iops/bps limit in different ways:
>
> 1) for iops limit, there is no flag to record if the bio is throttled,
> and iops is always applied.
> 2) for bps limit, original bio will be flagged with BIO_BPS_THROTTLED,
> and io throttle will ignore bio with the flag.
>
> Noted this patch also remove the code to set flag in __bio_clone(), it's
> introduced in commit 111be8839817 ("block-throttle: avoid double
> charge"), and author thinks split bio can be resubmited and throttled
> again, which is wrong because split bio will continue to dispatch from
> caller.
>
> Fixes: 9f5ede3c01f9 ("block: throttle split bio in case of iops limit")
> Cc: <stable@xxxxxxxxxxxxxxx>
> Signed-off-by: Yu Kuai <yukuai3@xxxxxxxxxx>
Acked-by: Tejun Heo <tj@xxxxxxxxxx>
Thanks.
--
tejun
