Hi Linus,
Here are the io_uring changes queued up for 5.13. This pull request
contains:
- Support for multi-shot mode for POLL requests
- More efficient reference counting. This is shamelessly stolen from
the mm side. Even though referencing is mostly single/dual user, the
128 count was retained to keep the code the same. Maybe this
should/could be made generic at some point.
- Removal of the need to have a manager thread for each ring. The manager
threads only job was checking and creating new io-threads as needed,
instead we handle this from the queue path.
- Allow SQPOLL without CAP_SYS_ADMIN or CAP_SYS_NICE. Since 5.12, this
thread is "just" a regular application thread, so no need to restrict
use of it anymore.
- Cleanup of how internal async poll data lifetime is managed.
- Fix for syzbot reported crash on SQPOLL cancelation.
- Make buffer registration more like file registrations, which includes
flexibility in avoiding full set unregistration and re-registration.
- Fix for io-wq affinity setting.
- Be a bit more defensive in task->pf_io_worker setup.
- Various SQPOLL fixes.
- Cleanup of SQPOLL creds handling.
- Improvements to in-flight request tracking.
- File registration cleanups.
- Tons of cleanups and little fixes
Please pull!
The following changes since commit d434405aaab7d0ebc516b68a8fc4100922d7f5ef:
Linux 5.12-rc7 (2021-04-11 15:16:13 -0700)
are available in the Git repository at:
git://git.kernel.dk/linux-block.git tags/for-5.13/io_uring-2021-04-27
for you to fetch changes up to 7b289c38335ec7bebe45ed31137d596c808e23ac:
io_uring: maintain drain logic for multishot poll requests (2021-04-27 07:38:58 -0600)
----------------------------------------------------------------
for-5.13/io_uring-2021-04-27
----------------------------------------------------------------
Bijan Mottahedeh (1):
io_uring: implement fixed buffers registration similar to fixed files
Colin Ian King (1):
io_uring: Fix uninitialized variable up.resv
Hao Xu (4):
io-wq: simplify code in __io_worker_busy()
io_uring: check sqring and iopoll_list before shedule
io_uring: update sq_thread_idle after ctx deleted
io_uring: maintain drain logic for multishot poll requests
Jens Axboe (26):
io_uring: wrap io_kiocb reference count manipulation in helpers
io_uring: switch to atomic_t for io_kiocb reference count
io_uring: don't check for io_uring_fops for fixed files
io_uring: cache async and regular file state for fixed files
io_uring: correct comment on poll vs iopoll
io_uring: transform ret == 0 for poll cancelation completions
io_uring: mask in error/nval/hangup consistently for poll
io_uring: include cflags in completion trace event
io_uring: add multishot mode for IORING_OP_POLL_ADD
io_uring: abstract out helper for removing poll waitqs/hashes
io_uring: terminate multishot poll for CQ ring overflow
io_uring: abstract out a io_poll_find_helper()
io_uring: allow events and user_data update of running poll requests
kernel: allow fork with TIF_NOTIFY_SIGNAL pending
io-wq: eliminate the need for a manager thread
io_uring: allow SQPOLL without CAP_SYS_ADMIN or CAP_SYS_NICE
io_uring: fix race around poll update and poll triggering
task_work: add helper for more targeted task_work canceling
io-wq: cancel task_work on exit only targeting the current 'wq'
io_uring: don't attempt re-add of multishot poll request if racing
io_uring: provide io_resubmit_prep() stub for !CONFIG_BLOCK
io_uring: disable multishot poll for double poll add cases
io_uring: put flag checking for needing req cleanup in one spot
io_uring: tie req->apoll to request lifetime
io_uring: fix merge error for async resubmit
io-wq: remove unused io_wqe_need_worker() function
Palash Oswal (1):
io_uring: Check current->io_uring in io_uring_cancel_sqpoll
Pavel Begunkov (120):
io_uring: avoid taking ctx refs for task-cancel
io_uring: reuse io_req_task_queue_fail()
io_uring: further deduplicate file slot selection
io_uring: add a helper failing not issued requests
io_uring: refactor provide/remove buffer locking
io_uring: use better types for cflags
io_uring: refactor out send/recv async setup
io_uring: untie alloc_async_data and needs_async_data
io_uring: rethink def->needs_async_data
io_uring: merge defer_prep() and prep_async()
io_uring: simplify io_resubmit_prep()
io_uring: simplify io_sqd_update_thread_idle()
io_uring: don't take ctx refs in task_work handler
io_uring: optimise io_uring_enter()
io_uring: optimise tctx node checks/alloc
io_uring: keep io_req_free_batch() call locality
io_uring: inline __io_queue_linked_timeout()
io_uring: optimise success case of __io_queue_sqe
io_uring: refactor io_flush_cached_reqs()
io_uring: refactor rsrc refnode allocation
io_uring: inline io_put_req and friends
io_uring: refactor io_free_req_deferred()
io_uring: add helper flushing locked_free_list
io_uring: remove __io_req_task_cancel()
io_uring: inline io_clean_op()'s fast path
io_uring: optimise io_dismantle_req() fast path
io_uring: abolish old io_put_file()
io_uring: optimise io_req_task_work_add()
io_uring: don't clear REQ_F_LINK_TIMEOUT
io_uring: don't do extra EXITING cancellations
io_uring: remove tctx->sqpoll
io-wq: refactor *_get_acct()
io_uring: don't init req->work fully in advance
io_uring: kill unused REQ_F_NO_FILE_TABLE
io_uring: optimise kiocb_end_write for !ISREG
io_uring: don't alter iopoll reissue fail ret code
io_uring: hide iter revert in resubmit_prep
io_uring: optimise rw complete error handling
io_uring: allocate memory for overflowed CQEs
io_uring: reg buffer overflow checks hardening
io_uring: name rsrc bits consistently
io_uring: simplify io_rsrc_node_ref_zero
io_uring: use rsrc prealloc infra for files reg
io_uring: encapsulate rsrc node manipulations
io_uring: move rsrc_put callback into io_rsrc_data
io_uring: refactor io_queue_rsrc_removal()
io_uring: ctx-wide rsrc nodes
io_uring: reuse io_rsrc_node_destroy()
io_uring: remove useless is_dying check on quiesce
io_uring: combine lock/unlock sections on exit
io_uring: better ref handling in poll_remove_one
io_uring: remove unused hash_wait
io_uring: refactor io_async_cancel()
io_uring: improve import_fixed overflow checks
io_uring: store reg buffer end instead of length
io_uring: kill unused forward decls
io_uring: lock annotate timeouts and poll
io_uring: simplify overflow handling
io_uring: put link timeout req consistently
io_uring: deduplicate NOSIGNAL setting
io_uring: set proper FFS* flags on reg file update
io_uring: don't quiesce intial files register
io_uring: refactor file tables alloc/free
io_uring: encapsulate fixed files into struct
io_uring: kill outdated comment about splice punt
io_uring: clean up io_poll_task_func()
io_uring: refactor io_poll_complete()
io_uring: simplify apoll hash removal
io_uring: unify task and files cancel loops
io_uring: track inflight requests through counter
io_uring: unify files and task cancel
io_uring: refactor io_close
io_uring: enable inline completion for more cases
io_uring: refactor compat_msghdr import
io_uring: optimise non-eventfd post-event
io_uring: always pass cflags into fill_event()
io_uring: optimise fill_event() by inlining
io_uring: simplify io_rsrc_data refcounting
io_uring: add buffer unmap helper
io_uring: cleanup buffer register
io_uring: split file table from rsrc nodes
io_uring: improve sqo stop
io_uring: improve hardlink code generation
io_uring: return back safer resurrect
io_uring: fix leaking reg files on exit
io_uring: fix uninit old data for poll event upd
io_uring: split poll and poll update structures
io_uring: add timeout completion_lock annotation
io_uring: refactor hrtimer_try_to_cancel uses
io_uring: clean up io_poll_remove_waitqs()
io_uring: don't fail overflow on in_idle
io_uring: skip futile iopoll iterations
io_uring: inline io_iopoll_getevents()
io_uring: refactor io_ring_exit_work()
io_uring: fix POLL_REMOVE removing apoll
io_uring: add helper for parsing poll events
io_uring: move poll update into remove not add
io_uring: don't fail submit with overflow backlog
io_uring: fix overflows checks in provide buffers
io_uring: check register restriction afore quiesce
io_uring: remove extra sqpoll submission halting
io_uring: fix shared sqpoll cancellation hangs
io_uring: move inflight un-tracking into cleanup
io_uring: safer sq_creds putting
io_uring: refactor io_sq_offload_create()
io_uring: move __io_sqe_files_unregister
io_uring: return back rsrc data free helper
io_uring: decouple CQE filling from requests
io_uring: preparation for rsrc tagging
io_uring: add generic path for rsrc update
io_uring: enumerate dynamic resources
io_uring: add IORING_REGISTER_RSRC
io_uring: add generic rsrc update with tags
io_uring: keep table of pointers to ubufs
io_uring: prepare fixed rw for dynanic buffers
io_uring: add full-fledged dynamic buffers support
io_uring: fix invalid error check after malloc
io_uring: fix work_exit sqpoll cancellations
io_uring: simplify SQPOLL cancellations
io_uring: fix NULL reg-buffer
Peter Zijlstra (1):
io-wq: Fix io_wq_worker_affinity()
Stefan Metzmacher (2):
kernel: always initialize task->pf_io_worker to NULL
io_uring: io_sq_thread() no longer needs to reset current->pf_io_worker
fs/io-wq.c | 336 +++--
fs/io-wq.h | 1 +
fs/io_uring.c | 2612 +++++++++++++++++++++------------------
include/linux/io_uring.h | 12 +-
include/linux/task_work.h | 2 +
include/trace/events/io_uring.h | 11 +-
include/uapi/linux/io_uring.h | 40 +
kernel/fork.c | 3 +-
kernel/task_work.c | 35 +-
9 files changed, 1599 insertions(+), 1453 deletions(-)
--
Jens Axboe
