On Tue, Mar 29, 2011 at 10:41:20AM +0800, Wu Fengguang wrote: > On Tue, Mar 29, 2011 at 10:14:58AM +0800, Dave Chinner wrote: > > -printable > > Content-Length: 2034 > > Lines: 51 > > > > On Mon, Mar 28, 2011 at 10:44:45AM +0800, Wu Fengguang wrote: > > > On Sat, Mar 26, 2011 at 07:05:44AM +0800, Jan Kara wrote: > > > > And actually the NFS traces you pointed to originally seem to be different > > > > problem, in fact not directly related to what balance_dirty_pages() does... > > > > And with local filesystem the results seem to be reasonable (although there > > > > are some longer sleeps in your JBOD measurements I don't understand yet). > > > > > > Yeah the NFS case can be improved on the FS side (for now you may just > > > reuse my NFS patches and focus on other generic improvements). > > > > > > The JBOD issue is also beyond my understanding. > > > > > > Note that XFS will also see one big IO completion per 0.5-1 seconds, > > > when we are to increase the write chunk size from the current 4MB to > > > near the bdi's write bandwidth. As illustrated by this graph: > > > > > > http://www.kernel.org/pub/linux/kernel/people/wfg/writeback/dirty-throttling-v6/4G/xfs-1dd-1M-8p-3927M-20%25-2.6.38-rc6-dt6+-2011-02-27-22-58/global_dirtied_written-500.png > > > > Which is _bad_. > > > > Increasing the writeback chunk size simply causes dirty queue > > starvation issues when there are lots of dirty files and lots more > > memory than there is writeback bandwidth. Think of a machine with > > 1TB of RAM (that's a 200GB dirty limit) and 1GB/s of disk > > throughput. Thats 3 minutes worth of writeback and increasing the > > chunk size to ~1s worth of throughput means that the 200th dirty > > file won't get serviced for 3 minutes.... > > > > We used to have behaviour similar to this this (prior to 2.6.16, IIRC), > > and it caused all sorts of problems where people were losing 10-15 > > minute old data when the system crashed because writeback didn't > > process the dirty inode list fast enough in the presence of lots of > > large files.... > > Yes it is a problem, and can be best solved by automatically lowering > bdi dirty limit to (bdi->write_bandwidth * dirty_expire_interval/100). > Then we reliably control the lost data size to < 30s by default. Perhaps, though I see problems with that also. e.g. write bandwidth is 100MB/s (dirty limit ~= 3GB), then someone runs a find on the same disk and write bandwidth drops to 10MB/s (dirty limit changes to ~300MB). Then we are 10x over the new dirty limit and the writing application will be completely throttled for the next 270s until the dirty pages drop below the new dirty limit or the find stops. IOWs, it changing IO workloads will cause interesting corner cases to be discovered and hence further complexity to handle effectively. And trying to diagnose problems because of such changes in IO load will be nigh on impossible - how would you gather sufficient information to determine that application A stalled for a minute because application B read a bunch of stuff from disk at the wrong time? Then how would you prove that you'd fixed the problem without introducing some other regression triggered by different workload changes? > > A small writeback chunk size has no adverse impact on XFS as long as > > the elevator does it's job of merging IOs (which in 99.9% of cases > > it does) so I'm wondering what the reason for making this change > > is. > > It's explained in this changelog (is the XFS paragraph still valid?) > > https://patchwork.kernel.org/patch/605151/ You mean this paragraph? "According to Christoph, the current writeback size is way too small, and XFS had a hack that bumped out nr_to_write to four times the value sent by the VM to be able to saturate medium-sized RAID arrays. This value was also problematic for ext4 as well, as it caused large files to be come interleaved on disk by in 8 megabyte chunks (we bumped up the nr_to_write by a factor of two)." We _used_ to have such a hack. It was there from 2.6.30 through to 2.6.35 - from when we realised writeback had bitrotted into badness to when we fixed the last set of bugs that the nr_to_write windup was papering over. between 2.6.30 and 2.6.35 we changed to dedicated flusher threads, got rid of congestion backoff, fixed up a bunch of queueing issues and finally stopped nr_to_write from going and staying negative and getting stuck on single inodes until they had no more dirty pages left. That was when this was committed: commit 254c8c2dbf0e06a560a5814eb90cb628adb2de66 Author: Dave Chinner <dchinner@xxxxxxxxxx> Date: Wed Jun 9 10:37:19 2010 +1000 xfs: remove nr_to_write writeback windup. Now that the background flush code has been fixed, we shouldn't need to silently multiply the wbc->nr_to_write to get good writeback. Remove that code. Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx> Reviewed-by: Christoph Hellwig <hch@xxxxxx> Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> And writeback throughput is now as good as it ever was.... > The larger write chunk size generally helps ext4 and RAID setups. Is this still true with ext4's new submit_bio()-based writeback IO submission path that was copied from the XFS? It's a lot more efficient so should be much better on RAID setups. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
