On Tue, Jun 17, 2014 at 09:52:46AM +0200, Lukáš Czerner wrote: > On Mon, 16 Jun 2014, Darrick J. Wong wrote: > > > Date: Mon, 16 Jun 2014 12:20:09 -0700 > > From: Darrick J. Wong <darrick.wong@xxxxxxxxxx> > > To: Tanya Brokhman <tlinder@xxxxxxxxxxxxxx> > > Cc: linux-fsdevel@xxxxxxxxxxxxxxx, linux-ext4@xxxxxxxxxxxxxxx, > > kdorfman@xxxxxxxxxxxxxx, merez@xxxxxxxxxxxxxx, > > Dolev Raviv <draviv@xxxxxxxxxxxxxx> > > Subject: Re: Quadrant write performance degradation - kernel3.10 vs kernel3.4 > > > > On Mon, Jun 16, 2014 at 09:02:08AM +0300, Tanya Brokhman wrote: > > > Hello, > > > Recently we encountered a performance degradation on 3.10kernel > > > based build, compared to 3.4 based one, when running the fs_write > > > Quadrant benchmark. > > > We profiled the test and came to the conclusion that the root cause > > > of the degradation is in the vfs_write call stack (overhead of > > > 2611.2us is observed in 3.10 kernel compared to 3.4): > > > > > > ret_fast_syscall > > > SyS_write > > > vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us) > > > do_sync_write > > > ext4_file_write > > > generic_file_aio_write (total time spent: 3.10kernel-19124.4us, > > > 3.4kernel-16815us) > > > __generic_file_aio_write > > > generic_file_buffered_write > > > ext4_da_write_begin (total time spent: 3.10kernel-10935.2us, > > > 3.4kernel-8444.6us) > > > __block_write_begin > > > ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us, > > > 3.4kernel-3576.8us) > > > ext4_es_lookup_extent (total time spent: 3.10kernel-2219.7us, > > > 3.4kernel-0us) [snip] > > > 2) Extents status tracking: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20 > > > “There is a cache extent for write access, so if writes are not very > > > random, adding space operations are in O(1) time.” > > > > I'm no expert on the extent status cache, but this seems like a possible cause. > > Exactly, there has been some fixes since the introduction of extent > status tree, however I've noticed some performance going down as > well and I believe that extent status tree is to blame. > > AFAIK you can not turn it off in any way, but there might be some > way to test it's overhead. Zheng, do you have any suggestions ? Sigh, sorry for the delay reply. Lukas, Could you please share your test with me? From the calltrace it seems that the latency is in ext4_da_get_block_prep. It is not easy to disable ext4_es_lookup_extent() because we need to lookup delayed extent from extent status tree and determine whether or not we need to reserve some disk spaces. Tanya, I really appreciate if you can disable delalloc and re-run your test. You can use the following command to turn off the delalloc feature. $ sudo mount -t ext4 -o remount,nodelalloc ${DEV} ${MNT} Thanks, - Zheng > > Thanks! > -Lukas > > > > > --D > > > > > > We tried pick up several performance-enhancement patches from the > > > community, released between 3.10 and 3.14 kernel versions. The > > > performance was almost the same. > > > > > > I was wondering what performance tests were performed on these > > > features? Has anyone encountered same issue? > > > > > > Best Regards > > > Tanya Brokhman > > > -- > > > QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member > > > of Code Aurora Forum, hosted by The Linux Foundation > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
