Ceph users reported that when using Ceph on ext4, the filesystem
would often become corrupted, containing inodes with incorrect
i_blocks counters.
I managed to reproduce this with a very hacked-up "streamtest"
binary from the Ceph tree.
Ceph is doing a lot of xattr writes, to out-of-inode blocks.
There is also another thread which does sync_file_range and close,
of the same files. The problem appears to happen due to this race:
sync/flush thread xattr-set thread
----------------- ----------------
do_writepages ext4_xattr_set
ext4_da_writepages ext4_xattr_set_handle
mpage_da_map_blocks ext4_xattr_block_set
set DELALLOC_RESERVE
ext4_new_meta_blocks
ext4_mb_new_blocks
if (!i_delalloc_reserved_flag)
vfs_dq_alloc_block
ext4_get_blocks
down_write(i_data_sem)
set i_delalloc_reserved_flag
...
up_write(i_data_sem)
if (i_delalloc_reserved_flag)
vfs_dq_alloc_block_nofail
In other words, the sync/flush thread pops in and sets
i_delalloc_reserved_flag on the inode, which makes the xattr thread
think that it's in a delalloc path in ext4_new_meta_blocks(),
and add the block for a second time, after already having added
it once in the !i_delalloc_reserved_flag case in ext4_mb_new_blocks
I think this can be as simple as taking i_data_sem before we
go down the new metablock path, so that we won't be testing
i_delalloc_reserved_flag in a race with xt4_get_blocks setting it.
Signed-off-by: Eric Sandeen <sandeen@xxxxxxxxxx>
---
This seems like slight abuse of i_data_sem though, since we are
not modifying i_data[] in the inode on the xattr path. But nothing
else protects readers of i_delalloc_reserved_flag in other
threads. Thoughts?
p.s. this almost feels like it needs quite a rework, I'm trying
to remember why we set a "we are in a delalloc path" in an inode
flag rather than as something passed through the callchain
arguments, but I'm fuzzy on these twisty paths.
diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index e4d0fca..dcebdbd 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -819,8 +819,14 @@ inserted:
if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
goal = goal & EXT4_MAX_BLOCK_FILE_PHYS;
+ /*
+ * take i_data_sem because we will test
+ * i_delalloc_reserved_flag in ext4_mb_new_blocks */
+ */
+ down_read((&EXT4_I(inode)->i_data_sem));
block = ext4_new_meta_blocks(handle, inode,
goal, NULL, &error);
+ up_read((&EXT4_I(inode)->i_data_sem));
if (error)
goto cleanup;
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
