This is a note to let you know that I've just added the patch titled
io_uring: Use io_schedule* in cqring wait
to the 6.1-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
io_uring-use-io_schedule-in-cqring-wait.patch
and it can be found in the queue-6.1 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@xxxxxxxxxxxxxxx> know about it.
>From f5f24ec27340daf12177fd09c2d107a589cbf527 Mon Sep 17 00:00:00 2001
From: Andres Freund <andres@xxxxxxxxxxx>
Date: Sun, 16 Jul 2023 12:13:06 -0600
Subject: io_uring: Use io_schedule* in cqring wait
From: Andres Freund <andres@xxxxxxxxxxx>
Commit 8a796565cec3601071cbbd27d6304e202019d014 upstream.
I observed poor performance of io_uring compared to synchronous IO. That
turns out to be caused by deeper CPU idle states entered with io_uring,
due to io_uring using plain schedule(), whereas synchronous IO uses
io_schedule().
The losses due to this are substantial. On my cascade lake workstation,
t/io_uring from the fio repository e.g. yields regressions between 20%
and 40% with the following command:
./t/io_uring -r 5 -X0 -d 1 -s 1 -c 1 -p 0 -S$use_sync -R 0 /mnt/t2/fio/write.0.0
This is repeatable with different filesystems, using raw block devices
and using different block devices.
Use io_schedule_prepare() / io_schedule_finish() in
io_cqring_wait_schedule() to address the difference.
After that using io_uring is on par or surpassing synchronous IO (using
registered files etc makes it reliably win, but arguably is a less fair
comparison).
There are other calls to schedule() in io_uring/, but none immediately
jump out to be similarly situated, so I did not touch them. Similarly,
it's possible that mutex_lock_io() should be used, but it's not clear if
there are cases where that matters.
Cc: stable@xxxxxxxxxxxxxxx # 5.10+
Cc: Pavel Begunkov <asml.silence@xxxxxxxxx>
Cc: io-uring@xxxxxxxxxxxxxxx
Cc: linux-kernel@xxxxxxxxxxxxxxx
Signed-off-by: Andres Freund <andres@xxxxxxxxxxx>
Link: https://lore.kernel.org/r/20230707162007.194068-1-andres@xxxxxxxxxxx
[axboe: minor style fixup]
Signed-off-by: Jens Axboe <axboe@xxxxxxxxx>
Signed-off-by: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
---
io_uring/io_uring.c | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2346,7 +2346,7 @@ static inline int io_cqring_wait_schedul
struct io_wait_queue *iowq,
ktime_t *timeout)
{
- int ret;
+ int token, ret;
unsigned long check_cq;
/* make sure we run task_work before checking for signals */
@@ -2362,9 +2362,18 @@ static inline int io_cqring_wait_schedul
if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))
return -EBADR;
}
+
+ /*
+ * Use io_schedule_prepare/finish, so cpufreq can take into account
+ * that the task is waiting for IO - turns out to be important for low
+ * QD IO.
+ */
+ token = io_schedule_prepare();
+ ret = 1;
if (!schedule_hrtimeout(timeout, HRTIMER_MODE_ABS))
- return -ETIME;
- return 1;
+ ret = -ETIME;
+ io_schedule_finish(token);
+ return ret;
}
/*
Patches currently in stable-queue which might be from andres@xxxxxxxxxxx are
queue-6.1/io_uring-use-io_schedule-in-cqring-wait.patch
