1-4 kills REQ_F_COMPLETE_INLINE as we're out of bits.
Patch 5 from Hao should remove some overhead from poll requests
Patch 6 from Hao adds per-bucket spinlocks, and 16-19 do a little
bit of cleanup. The downside of per-bucket spinlocks is that it adds
additional spinlock/unlock pair in the poll request completion side,
which shouldn't matter much with 20/25.
Patch 11 uses inline completion infra for poll requests, this nicely
improves perf when there is a good tw batching.
Patch 12 implements the userspace visible side of
IORING_SETUP_SINGLE_ISSUER, it'll be used for poll requests and
later for spinlock optimisations.
13-16 introduces ->uring_lock protected cancellation hashing. It
requires us to grab ->uring_lock in the completion side, but saves
two spin lock/unlock pairs. We apply it automatically in cases the
mutex is already likely to be held (see 25/25 description), so there
is no additional mutex overhead and potential latency problemes.
Numbers:
The used poll benchmark each iteration queues a batch of 32 POLLIN
poll requests and triggers all of them with read (+write).
baseline (patches 1-10):
11720 K req/s
base + 11 (+ inline completion infra)
12419 K req/s, ~+6%
base + 11-16 (+ uring_lock hashing):
12804 K req/s, +9.2% from the baseline, or +3.2% relative to patch 19.
Note that patch 11 only helps performance of poll-add requests, whenever
16/16 also improves apoll.
v2:
don't move ->cancel_seq out of iowq work struct
fix up single-issuer
v3:
clarify locking expectation around ->uring_lock hashing
don't complete by hand in io_read/write (see 1/16)
Hao Xu (2):
io_uring: poll: remove unnecessary req->ref set
io_uring: switch cancel_hash to use per entry spinlock
Pavel Begunkov (14):
io_uring: rw: delegate sync completions to core io_uring
io_uring: kill REQ_F_COMPLETE_INLINE
io_uring: refactor io_req_task_complete()
io_uring: don't inline io_put_kbuf
io_uring: pass poll_find lock back
io_uring: clean up io_try_cancel
io_uring: limit the number of cancellation buckets
io_uring: clean up io_ring_ctx_alloc
io_uring: use state completion infra for poll reqs
io_uring: add IORING_SETUP_SINGLE_ISSUER
io_uring: pass hash table into poll_find
io_uring: introduce a struct for hash table
io_uring: propagate locking state to poll cancel
io_uring: mutex locked poll hashing
include/uapi/linux/io_uring.h | 5 +-
io_uring/cancel.c | 23 +++-
io_uring/cancel.h | 4 +-
io_uring/fdinfo.c | 11 +-
io_uring/io_uring.c | 84 ++++++++-----
io_uring/io_uring.h | 5 -
io_uring/io_uring_types.h | 21 +++-
io_uring/kbuf.c | 33 +++++
io_uring/kbuf.h | 38 +-----
io_uring/poll.c | 225 +++++++++++++++++++++++++---------
io_uring/poll.h | 3 +-
io_uring/rw.c | 41 +++----
io_uring/tctx.c | 27 +++-
io_uring/tctx.h | 4 +-
io_uring/timeout.c | 3 +-
15 files changed, 353 insertions(+), 174 deletions(-)
--
2.36.1
