A few fixes and cleanups for the core part of this, and then a few experimental patches at the end: 1) Add support for polling for the SQTHREAD. If enabled, this will poll the SQ ring for submissions, and (if polling is enabled), it will poll for completions and place them in the CQ ring. This means an application will never have to enter the kernel do do IO, it simply updates the SQ ring with what it needs. Currently the thread goes to sleep after an idle second, to avoid spinning all the time. But if IO is continually flowing, it will keep running. Some room for improvement here, with some sort of backoff in terms of spinning and sleeping, but the concept seems sound. When updating the SQ ring with new entries, the application needs to do: barrier(); if (sq_ring->kflags & IORING_SQ_NEED_WAKEUP) io_ring_enter(ctx, to_submit, 0, IORING_SUBMIT); in case the thread has gone to sleep. If IORING_SQ_NEED_WAKEUP isn't set, then no call needs to be made. This works both with polled IO, in which case the SQ thread will poll for completions too (as mentioned), and with buffered/O_DIRECT async aio. For the latter, completions end up in the CQ ring when the IRQ triggers. 2) Utilization of struct io_event ->res2. This is unused in the kernel. One idea is to populate this with meta data about the IO operation. For buffered aio, we can tell the application if the IO hit media or was served out of the page cache. These kinds of metrics may be useful for an application. For the latter, the sample application I wrote: http://git.kernel.dk/cgit/fio/plain/t/aio-ring.c will give you by-the-second cache hit rates when run with buffered=1, and fio with the aio-ring engine will do the same. As before, find these patches in my aio-poll branch: http://git.kernel.dk/cgit/linux-block/log/?h=aio-poll or clone it here: git://git.kernel.dk/linux-block aio-poll and the code is based on my for-4.21/aio branch, which holds the aio patches queued up for 4.21 already. Changes since v8: - Get rid of kmalloc/kfree for inline cached buffered aio - Add IOCTX_FLAG_SQPOLL - Correct syscall prototype in <linux/syscalls.h> to match actual syscall - Add "IO was cached" hint to res2 - Fixed an issue with indexing fixed buffer mappings - Check for fixed buffer location, EFAULT if app changed location - Unify bio_iov_bvec_add_pages() and bio_iov_iter_get_pages() - Split out prep patches - Style fixups Documentation/filesystems/vfs.txt | 3 + Documentation/sysctl/fs.txt | 8 +- arch/x86/entry/syscalls/syscall_64.tbl | 2 + block/bio.c | 56 +- fs/aio.c | 1938 ++++++++++++++++++++++-- fs/block_dev.c | 23 +- fs/file.c | 15 +- fs/file_table.c | 9 +- fs/gfs2/file.c | 2 + fs/iomap.c | 52 +- fs/xfs/xfs_file.c | 1 + include/linux/blk_types.h | 2 + include/linux/file.h | 2 + include/linux/fs.h | 6 +- include/linux/iomap.h | 1 + include/linux/syscalls.h | 4 + include/uapi/asm-generic/unistd.h | 4 +- include/uapi/linux/aio_abi.h | 47 + kernel/sys_ni.c | 2 + 19 files changed, 1993 insertions(+), 184 deletions(-) -- Jens Axboe
