I've found that write performance of exFAT is a lot lower than other
filesystems on our setup with a fast NVMe SSD:
- Kernel: 6.10.0-364.vanilla.fc40.x86_64
- Test cmd: fio -name=fioWr -ioengine=libaio -directory=<mount point>
-blocksize=1M -readwrite=write -filesize=10G -end_fsync=1 -numjobs=8
-direct=1 -group_reporting
- Benchmarks:
- exFAT: Direct I/O: ~1370 MiB/s (Buffered: ~1250 MiB/s)
(mkfs.exfat -c 1M -b 1M <device>)
- ext4: Direct I/O: ~2230 MiB/s (Buffered: ~2150 MiB/s)
- xfs: Direct I/O: ~2220 MiB/s (Buffered: ~2200 MiB/s)
I found that direct I/O is disabled for most of these writes in exFAT,
code responsible is "exfat_direct_IO()" in "fs/exfat/inode.c". As the
written file is being expanded it is falling back to normal buffered
write. The relevant code also has the following FIXME comment
(inherited from the fat driver):
/*
* FIXME: blockdev_direct_IO() doesn't use ->write_begin(),
* so we need to update the ->i_size_aligned to block boundary.
*
* But we must fill the remaining area or hole by nul for
* updating ->i_size_aligned
*
* Return 0, and fallback to normal buffered write.
*/
I have tried working around this problem by preallocating the file. I
see good write speeds in the direct I/O case (matching ext4 ~2200
MiB/s, Buffered: ~1700 MiB/s), the problem is that preallocation is
slow since native fallocate is not supported.
What would the maintainers recommend as the best course of action for
us here? I have noticed [1] that FALLOC_FL_KEEP_SIZE was implemented
in fat, so perhaps that would be simple to work on first?
Does anyone with more experience in the exFAT driver know the full
reasons behind the direct I/O disabling, and what it would take to
support direct I/O during file extension? Perhaps recent changes may
have made fixing this simpler?
Thanks.
1: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/fat?id=b13bb33eacb7266d66a3adf03adaa0886d091789
