hi,
Hi Xiaoguang,
Why exactly you don't get the performance [1] is referring to could have many reasons.
Yes, I think so, and I'll keep looking into it.
It seems your nop test code quite different from the echo server that is used for [1]. [1] employs and event loop (single thread) specifically for networking. It saves an expensive call to epoll in each iteration of the loop.
This nop testcase is just writtent to evaluate io_uring framework.
I have added codes to count the number of syscalls in io_uring_echo_server.c and
io_uring really reduces syscalls tremendously, so that's my question that why I still
don't see the obvious improvements though io_uring reduces many syscalls, then I wrote
the mail to discuss io_uring framework's self overhead :)
I glanced at your test program, what is the performance difference with IOSQE_ASYNC vs without IOSQE_ASYNC?
If IOSQE_ASYNC is flagged, io_uring will complete nop operations in io_uring
internal thread pool, so it'll be a bit slower.
[1]: https://github.com/frevib/io_uring-echo-server/blob/io-uring-feat-fast-poll/benchmarks/benchmarks.md.
Do you have free time to have a test in newest kernel upstream in your test environment?
Or do you still remember the head kernel commit you tested?
And in your io-uring-feat-fast-poll branch, you use:
// tell kernel we have put a sqe on the submission ring
io_uring_submit(&ring);
// wait for new cqe to become available
ret = io_uring_wait_cqe(&ring, &cqe);
if (ret != 0)
{
perror("Error io_uring_wait_cqe\n");
exit(1);
}
Indeed you can use:
ret = io_uring_submit_and_wait(&ring, 1);
if (ret < 0) {
perror("Error io_uring_wait_cqe\n");
exit(1);
}
then we can reduce syscall more.
Regards,
Xiaoguang Wang
--
Hielke de Vries
On Fri, May 8, 2020, at 18:37, Jens Axboe wrote:
On 5/8/20 9:18 AM, Xiaoguang Wang wrote:
hi,
This issue was found when I tested IORING_FEAT_FAST_POLL feature, with
the newest upstream codes, indeed I find that io_uring's performace
improvement is not obvious compared to epoll in my test environment,
most of the time they are similar. Test cases basically comes from:
https://github.com/frevib/io_uring-echo-server/blob/io-uring-feat-fast-poll/benchmarks/benchmarks.md.
In above url, the author's test results shows that io_uring will get a
big performace improvement compared to epoll. I'm still looking into
why I don't get the big improvement, currently don't know why, but I
find some obvious regression issue.
I wrote a simple tool based io_uring nop operation to evaluate
io_uring framework in v5.1 and 5.7.0-rc4+(jens's io_uring-5.7 branch),
I see a obvious performace regression:
v5.1 kernel:
$sudo taskset -c 60 ./io_uring_nop_stress -r 300 # run 300 seconds
total ios: 1832524960
IOPS: 6108416
5.7.0-rc4+
$sudo taskset -c 60 ./io_uring_nop_stress -r 300
total ios: 1597672304
IOPS: 5325574
it's about 12% performance regression.
For sure there's a bit more bloat in 5.7+ compared to the initial slim
version, and focus has been on features to a certain extent recently.
The poll rework for 5.7 will really improve performance for the
networked side though, so it's not like it's just piling on features
that add bloat.
That said, I do think it's time for a revisit on overhead. It's been a
while since I've done my disk IO testing. The nop testing isn't _that_
interesting by itself, as a micro benchmark it does yield some results
though. Are you running on bare metal or in a VM?
Using perf can see many performance bottlenecks, for example,
io_submit_sqes is one. For now, I did't make many analysis yet, just
have a look at io_submit_sqes(), there are many assignment operations
in io_init_req(), but I'm not sure whether they are all needed when
req is not needed to be punt to io-wq, for example,
INIT_IO_WORK(&req->work, io_wq_submit_work); # a whole struct
assignment from perf annotate tool, it's an expensive operation, I
think reqs that use fast poll feature use task-work function, so the
INIT_IO_WORK maybe not necessary.
I'm sure there's some low hanging fruit there, and I'd love to take
patches for it.
Above is just one issue, what I worry is that whether io_uring is
becoming more bloated gradually, and will not that better to aio. In
https://kernel.dk/io_uring.pdf, it says that io_uring will eliminate
104 bytes copy compared to aio, but see currenct io_init_req(),
io_uring maybe copy more, introducing more overhead? Or does we need
to carefully re-design struct io_kiocb, to reduce overhead as soon as
possible.
The copy refers to the data structures coming in and out, both io_uring
and io_uring inititalize their main io_kiocb/aio_kiocb structure as
well. The io_uring is slightly bigger, but not much, and it's the same
number of cachelines. So should not be a huge difference there. The
copying on the aio side is basically first the pointer copy, then the
user side kiocb structure. io_uring doesn't need to do that. The
completion side is also slimmer. We also don't need as many system calls
to do the same thing, for example.
So no, we should always been substantially slimmer than aio, just by the
very nature of the API.
One major thing I've been thinking about for io_uring is io_kiocb
recycling. We're hitting the memory allocator for alloc+free for each
request, even though that can be somewhat amortized by doing batched
submissions, and polling for instance can also do batched frees. But I'm
pretty sure we can find some gains here by having some io_kiocb caching
that is persistent across operations.
Outside of that, runtime analysis and speeding up the normal path
through io_uring will probably also easily yield us an extra 5% (or
more).
--
Jens Axboe
