iomap_zero_range() does not correctly handle unwritten mappings with
dirty folios in pagecache. It skips unwritten mappings
unconditionally as if they were already zeroed, and thus potentially
exposes stale data from a previous write if affected folios are not
written back before the zero range.
Most callers already flush the target range of the zero for
unrelated, context specific reasons, so this problem is not
necessarily prevalent. The known outliers (in XFS) are file
extension via buffered write and truncate. The truncate path issues
a flush to work around this iomap problem, but the file extension
path does not and thus can expose stale data if current EOF is
unaligned and has a dirty folio over an unwritten block.
This patch implements a mechanism for making zero range pagecache
aware for filesystems that support mapping validation (i.e.
folio_ops->iomap_valid()). Instead of just skipping unwritten
mappings, scan the corresponding pagecache range for dirty or
writeback folios. If found, explicitly zero them via buffered write.
Clean or uncached subranges of unwritten mappings are skipped, as
before.
The quirk with a post-iomap_begin() pagecache scan is that it is
racy with writeback and reclaim activity. Even if the higher level
code holds the invalidate lock, nothing prevents a dirty folio from
being written back, cleaned, and even reclaimed sometime after
iomap_begin() returns an unwritten map but before a pagecache scan
might find the dirty folio. To handle this situation, we can rely on
the fact that writeback completion converts unwritten extents in the
fs before writeback state is cleared on the folio.
This means that a pagecache scan followed by a mapping revalidate of
an unwritten mapping should either find a dirty folio if it exists,
or detect a mapping change if a dirty folio did exist and had been
cleaned sometime before the scan but after the unwritten mapping was
found. If the revalidation succeeds then we can safely assume
nothing has been written back and skip the range. If the
revalidation fails then we must assume any offset in the range could
have been modified by writeback. In other words, we must be
particularly careful to make sure that any uncached range we intend
to skip does not make it into iter.processed until the mapping is
revalidated.
Altogether, this allows zero range to handle dirty folios over
unwritten extents without needing to flush and wait for writeback
completion.
Signed-off-by: Brian Foster <bfoster@xxxxxxxxxx>
---
fs/iomap/buffered-io.c | 53 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 50 insertions(+), 3 deletions(-)
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index a9425170df72..ea1d396ef445 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -1385,6 +1385,23 @@ iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
}
EXPORT_SYMBOL_GPL(iomap_file_unshare);
+/*
+ * Scan an unwritten mapping for dirty pagecache and return the length of the
+ * clean or uncached range leading up to it. This is the range that zeroing may
+ * skip once the mapping is validated.
+ */
+static inline loff_t
+iomap_zero_iter_unwritten(struct iomap_iter *iter, loff_t pos, loff_t length)
+{
+ struct address_space *mapping = iter->inode->i_mapping;
+ loff_t fpos = pos;
+
+ if (!filemap_range_has_writeback(mapping, &fpos, length))
+ return length;
+ /* fpos can be smaller if the start folio is dirty */
+ return max(fpos, pos) - pos;
+}
+
static loff_t iomap_zero_iter(struct iomap_iter *iter, bool *did_zero)
{
const struct iomap *srcmap = iomap_iter_srcmap(iter);
@@ -1393,16 +1410,46 @@ static loff_t iomap_zero_iter(struct iomap_iter *iter, bool *did_zero)
loff_t written = 0;
/* already zeroed? we're done. */
- if (srcmap->type == IOMAP_HOLE || srcmap->type == IOMAP_UNWRITTEN)
+ if (srcmap->type == IOMAP_HOLE)
return length;
do {
struct folio *folio;
int status;
size_t offset;
- size_t bytes = min_t(u64, SIZE_MAX, length);
+ size_t bytes;
+ loff_t pending = 0;
bool ret;
+ /*
+ * Determine the range of the unwritten mapping that is clean in
+ * pagecache. We can skip this range, but only if the mapping is
+ * still valid after the pagecache scan. This is because
+ * writeback may have cleaned folios after the mapping lookup
+ * but before we were able to find them here. If that occurs,
+ * then the mapping must now be stale and we must reprocess the
+ * range.
+ */
+ if (srcmap->type == IOMAP_UNWRITTEN) {
+ pending = iomap_zero_iter_unwritten(iter, pos, length);
+ if (pending == length) {
+ /* no dirty cache, revalidate and bounce as we're
+ * either done or the mapping is stale */
+ if (iomap_revalidate(iter))
+ written += pending;
+ break;
+ }
+
+ /*
+ * Found a dirty folio. Update pos/length to point at
+ * it. written is updated only after the mapping is
+ * revalidated by iomap_write_begin().
+ */
+ pos += pending;
+ length -= pending;
+ }
+
+ bytes = min_t(u64, SIZE_MAX, length);
status = iomap_write_begin(iter, pos, bytes, &folio);
if (status)
return status;
@@ -1422,7 +1469,7 @@ static loff_t iomap_zero_iter(struct iomap_iter *iter, bool *did_zero)
pos += bytes;
length -= bytes;
- written += bytes;
+ written += bytes + pending;
} while (length > 0);
if (did_zero)
--
2.45.0
