From: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
Prefaulting the write source buffer incurs an extra userspace access
in the common fast path. Make iomap_write_iter() consistent with
generic_perform_write(): only touch userspace an extra time when
copy_folio_from_iter_atomic() has failed to make progress.
Signed-off-by: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
Cc: Christian Brauner <brauner@xxxxxxxxxx>
Cc: "Darrick J. Wong" <djwong@xxxxxxxxxx>
Cc: linux-xfs@xxxxxxxxxxxxxxx
---
b/fs/iomap/buffered-io.c | 24 +++++++++---------------
1 file changed, 9 insertions(+), 15 deletions(-)
diff -puN fs/iomap/buffered-io.c~iomap-postfault fs/iomap/buffered-io.c
--- a/fs/iomap/buffered-io.c~iomap-postfault 2025-01-29 09:03:32.179361299 -0800
+++ b/fs/iomap/buffered-io.c 2025-01-29 09:03:32.187361965 -0800
@@ -937,21 +937,6 @@ retry:
if (bytes > length)
bytes = length;
- /*
- * Bring in the user page that we'll copy from _first_.
- * Otherwise there's a nasty deadlock on copying from the
- * same page as we're writing to, without it being marked
- * up-to-date.
- *
- * For async buffered writes the assumption is that the user
- * page has already been faulted in. This can be optimized by
- * faulting the user page.
- */
- if (unlikely(fault_in_iov_iter_readable(i, bytes) == bytes)) {
- status = -EFAULT;
- break;
- }
-
status = iomap_write_begin(iter, pos, bytes, &folio);
if (unlikely(status)) {
iomap_write_failed(iter->inode, pos, bytes);
@@ -1005,6 +990,15 @@ retry:
bytes = copied;
goto retry;
}
+ /*
+ * 'folio' is now unlocked and faults on it can be
+ * handled. Ensure forward progress by trying to
+ * fault it in now.
+ */
+ if (fault_in_iov_iter_readable(i, bytes) == bytes) {
+ status = -EFAULT;
+ break;
+ }
} else {
pos += written;
total_written += written;
_
