Quadrant write performance degradation - kernel3.10 vs kernel3.4

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,
Recently we encountered a performance degradation on 3.10kernel based build, compared to 3.4 based one, when running the fs_write Quadrant benchmark. We profiled the test and came to the conclusion that the root cause of the degradation is in the vfs_write call stack (overhead of 2611.2us is observed in 3.10 kernel compared to 3.4):

ret_fast_syscall
SyS_write
vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us)
do_sync_write
ext4_file_write
generic_file_aio_write (total time spent: 3.10kernel-19124.4us, 3.4kernel-16815us)
__generic_file_aio_write
generic_file_buffered_write
ext4_da_write_begin (total time spent: 3.10kernel-10935.2us, 3.4kernel-8444.6us)
__block_write_begin
ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us, 3.4kernel-3576.8us) ext4_es_lookup_extent (total time spent: 3.10kernel-2219.7us, 3.4kernel-0us)


We tried to revert just the ext4 code back to 3.4 (on a 3.10 kernel) build and got an improvement of 50% in the test result. When looking deeper into the changes made to the ext4 FS between 3.4 and 3.10 versions we stumbled across two major features making an explicit tradeoff in favor of robustness and good design over performance in some use cases:

1) Metadata Checksums http://kernelnewbies.org/Linux_3.5#head-e8ea0d70436ea63590eac3dc25a7b417333147f8 “As far as performance impact goes, it shouldn't be noticeable for common desktop and server workloads. A mail server ffsb simulation show nearly no change. On a test doing only file creation and deletion and extent tree modifications, a performance drop of about 20 percent was measured. However, it's a workload very heavily oriented towards metadata, in most real-world workloads metadata is usually a small fraction of total IO, so unless your workload is metadata-oriented, the cost of enabling this feature should be negligible.”

2) Extents status tracking: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20 “There is a cache extent for write access, so if writes are not very random, adding space operations are in O(1) time.”

We tried pick up several performance-enhancement patches from the community, released between 3.10 and 3.14 kernel versions. The performance was almost the same.

I was wondering what performance tests were performed on these features? Has anyone encountered same issue?

Best Regards
Tanya Brokhman
--
QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux