Hello,
Recently we encountered a performance degradation on 3.10kernel based
build, compared to 3.4 based one, when running the fs_write Quadrant
benchmark.
We profiled the test and came to the conclusion that the root cause of
the degradation is in the vfs_write call stack (overhead of 2611.2us is
observed in 3.10 kernel compared to 3.4):
ret_fast_syscall
SyS_write
vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us)
do_sync_write
ext4_file_write
generic_file_aio_write (total time spent: 3.10kernel-19124.4us,
3.4kernel-16815us)
__generic_file_aio_write
generic_file_buffered_write
ext4_da_write_begin (total time spent: 3.10kernel-10935.2us,
3.4kernel-8444.6us)
__block_write_begin
ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us,
3.4kernel-3576.8us)
ext4_es_lookup_extent (total time spent: 3.10kernel-2219.7us,
3.4kernel-0us)
We tried to revert just the ext4 code back to 3.4 (on a 3.10 kernel)
build and got an improvement of 50% in the test result.
When looking deeper into the changes made to the ext4 FS between 3.4 and
3.10 versions we stumbled across two major features making an explicit
tradeoff in favor of robustness and good design over performance in some
use cases:
1) Metadata Checksums
http://kernelnewbies.org/Linux_3.5#head-e8ea0d70436ea63590eac3dc25a7b417333147f8
“As far as performance impact goes, it shouldn't be noticeable for
common desktop and server workloads. A mail server ffsb simulation show
nearly no change. On a test doing only file creation and deletion and
extent tree modifications, a performance drop of about 20 percent was
measured. However, it's a workload very heavily oriented towards
metadata, in most real-world workloads metadata is usually a small
fraction of total IO, so unless your workload is metadata-oriented, the
cost of enabling this feature should be negligible.”
2) Extents status tracking:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20
“There is a cache extent for write access, so if writes are not very
random, adding space operations are in O(1) time.”
We tried pick up several performance-enhancement patches from the
community, released between 3.10 and 3.14 kernel versions. The
performance was almost the same.
I was wondering what performance tests were performed on these features?
Has anyone encountered same issue?
Best Regards
Tanya Brokhman
--
QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
