On Mon, Jun 16, 2014 at 09:02:08AM +0300, Tanya Brokhman wrote: > Hello, > Recently we encountered a performance degradation on 3.10kernel > based build, compared to 3.4 based one, when running the fs_write > Quadrant benchmark. > We profiled the test and came to the conclusion that the root cause > of the degradation is in the vfs_write call stack (overhead of > 2611.2us is observed in 3.10 kernel compared to 3.4): > > ret_fast_syscall > SyS_write > vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us) > do_sync_write > ext4_file_write > generic_file_aio_write (total time spent: 3.10kernel-19124.4us, > 3.4kernel-16815us) > __generic_file_aio_write > generic_file_buffered_write > ext4_da_write_begin (total time spent: 3.10kernel-10935.2us, > 3.4kernel-8444.6us) > __block_write_begin > ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us, > 3.4kernel-3576.8us) > ext4_es_lookup_extent (total time spent: 3.10kernel-2219.7us, > 3.4kernel-0us) > > > We tried to revert just the ext4 code back to 3.4 (on a 3.10 kernel) > build and got an improvement of 50% in the test result. > When looking deeper into the changes made to the ext4 FS between 3.4 > and 3.10 versions we stumbled across two major features making an > explicit tradeoff in favor of robustness and good design over > performance in some use cases: > > 1) Metadata Checksums http://kernelnewbies.org/Linux_3.5#head-e8ea0d70436ea63590eac3dc25a7b417333147f8 > “As far as performance impact goes, it shouldn't be noticeable for > common desktop and server workloads. A mail server ffsb simulation > show nearly no change. On a test doing only file creation and > deletion and extent tree modifications, a performance drop of about > 20 percent was measured. However, it's a workload very heavily > oriented towards metadata, in most real-world workloads metadata is > usually a small fraction of total IO, so unless your workload is > metadata-oriented, the cost of enabling this feature should be > negligible.” Dumb question, but do you have metadata_csum enabled? That would be a little surprising, since (afaik) the only way you can turn it on is via unreleased e2fsprogs-1.43. (Otoh if you /do/ have it enabled and it's slowing you down, I'd like to hear about it. ;)) > 2) Extents status tracking: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20 > “There is a cache extent for write access, so if writes are not very > random, adding space operations are in O(1) time.” I'm no expert on the extent status cache, but this seems like a possible cause. --D > > We tried pick up several performance-enhancement patches from the > community, released between 3.10 and 3.14 kernel versions. The > performance was almost the same. > > I was wondering what performance tests were performed on these > features? Has anyone encountered same issue? > > Best Regards > Tanya Brokhman > -- > QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member > of Code Aurora Forum, hosted by The Linux Foundation > -- > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
