This patch series adds support for async buffered writes when using both
xfs and io-uring. Currently io-uring only supports buffered writes in the
slow path, by processing them in the io workers. With this patch series it is
now possible to support buffered writes in the fast path. To be able to use
the fast path the required pages must be in the page cache, the required locks
in xfs can be granted immediately and no additional blocks need to be read
form disk.
Updating the inode can take time. An optimization has been implemented for
the time update. Time updates will be processed in the slow path. While there
is already a time update in process, other write requests for the same file,
can skip the update of the modification time.
Performance results:
For fio the following results have been obtained with a queue depth of
1 and 4k block size (runtime 600 secs):
sequential writes:
without patch with patch
iops: 80k 269k
random writes:
without patch with patch
iops: 76k 249k
For an io depth of 1, the new patch improves throughput by over three times
(compared to the exiting behavior, where buffered writes are processed by an
io-worker process) and also the latency is considerably reduced. To achieve the
same or better performance with the exisiting code an io depth of 4 is required.
Increasing the iodepth further does not lead to any further improvements.
Especially for mixed workloads this is a considerable improvement.
Support for async buffered writes:
Patch 1: block: add check for async buffered writes to generic_write_checks
Add a new flag FMODE_BUF_WASYNC so filesystems can specify that they support
async buffered writes and include the flag in the check of the function
generic_write_checks().
Patch 2: mm: add FGP_ATOMIC flag to __filemap_get_folio()
This adds the FGP_ATOMIC flag. This allows to specify the gfp flags
for memory allocations for async buffered writes.
Patch 3: iomap: add iomap_page_create_gfp to allocate iomap_pages
Add new function to allow specifying gfp flags when allocating and
initializing the structure iomap_page.
Patch 4: iomap: use iomap_page_create_gfp() in __iomap_write_begin
Add a gfp flag to the iomap_page_create function.
Patch 5: iomap: add async buffered write support
Set IOMAP_NOWAIT flag if IOCB_NOWAIT is set. Also use specific gfp flags if
the iomap_page structure is allocated for an async buffered page.
Patch 6: xfs: add iomap async buffered write support
Add async buffered write support to the xfs iomap layer.
Support for async buffered write support and inode time modification
Patch 7: fs: split off need_remove_file_privs() do_remove_file_privs()
Splits of a check and action function, so they can later be invoked
in the nowait code path.
Patch 8: fs: split off need_file_update_time and do_file_update_time
Splits of a check and action function, so they can later be invoked
in the nowait code path.
Patch 9: fs: add pending file update time flag.
Add new flag so consecutive write requests for the same inode can
proceed without waiting for the inode modification time to complete.
Patch 10: xfs: enable async write file modification handling.
Enable async write handling in xfs for the file modification time
update. If the file modification update requires logging or removal
of privileges that needs to wait, -EAGAIN is returned.
Patch 11: xfs: add async buffered write support
Take the ilock in nowait mode if async buffered writes are enabled.
Patch 12: io_uring: add support for async buffered writes
This enables the async buffered writes optimization in io_uring.
Buffered writes are enabled for blocks that are already in the page
cache.
Patch 13: io_uring: Add tracepoint for short writes
Support for write throttling of async buffered writes:
Patch 14: sched: add new fields to task_struct
Add two new fields to the task_struct. These fields store the
deadline after which writes are no longer throttled.
Patch 15: mm: support write throttling for async buffered writes
This changes the balance_dirty_pages function to take an additional
parameter. When nowait is specified the write throttling code no
longer waits synchronously for the deadline to expire. Instead
it sets the fields in task_struct. Once the deadline expires the
fields are reset.
Patch 16: iomap: User throttling for async buffered writes.
Enable async buffered write throttling in iomap.
Patch 17: io_uring: support write throttling for async buffered writes
Adds support to io_uring for write throttling. When the writes
are throttled, the write requests are added to the pending io list.
Once the write throttling deadline expires, the writes are submitted.
Enable async buffered write support in xfs
Patch 18: xfs: enable async buffered write support
This enables the flag that enables async buffered writes for xfs.
Testing:
This patch has been tested with xfstests and fio.
Stefan Roesch (18):
block: add check for async buffered writes to generic_write_checks
mm: add FGP_ATOMIC flag to __filemap_get_folio()
iomap: add iomap_page_create_gfp to allocate iomap_pages
iomap: use iomap_page_create_gfp() in __iomap_write_begin
iomap: add async buffered write support
xfs: add iomap async buffered write support
fs: split off need_remove_file_privs() do_remove_file_privs()
fs: split off need_file_update_time and do_file_update_time
fs: add pending file update time flag.
xfs: Enable async write file modification handling.
xfs: add async buffered write support
io_uring: add support for async buffered writes
io_uring: add tracepoint for short writes
sched: add new fields to task_struct
mm: support write throttling for async buffered writes
iomap: User throttling for async buffered writes.
io_uring: support write throttling for async buffered writes
xfs: enable async buffered write support
fs/inode.c | 127 +++++++++++++++++++++----------
fs/io_uring.c | 131 +++++++++++++++++++++++++++++---
fs/iomap/buffered-io.c | 63 +++++++++++++--
fs/read_write.c | 3 +-
fs/xfs/xfs_file.c | 51 +++++++++++--
fs/xfs/xfs_iomap.c | 33 +++++++-
include/linux/fs.h | 14 ++++
include/linux/pagemap.h | 1 +
include/linux/sched.h | 3 +
include/linux/writeback.h | 1 +
include/trace/events/io_uring.h | 25 ++++++
kernel/fork.c | 1 +
mm/filemap.c | 3 +
mm/page-writeback.c | 54 +++++++++----
14 files changed, 424 insertions(+), 86 deletions(-)
base-commit: af2d861d4cd2a4da5137f795ee3509e6f944a25b
--
2.30.2
