Re: [PATCH RFC 0/5] IO-less balance_dirty_pages() v2 (simple approach)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Mar 29, 2011 at 10:41:20AM +0800, Wu Fengguang wrote:
> On Tue, Mar 29, 2011 at 10:14:58AM +0800, Dave Chinner wrote:
> > -printable
> > Content-Length: 2034
> > Lines: 51
> > 
> > On Mon, Mar 28, 2011 at 10:44:45AM +0800, Wu Fengguang wrote:
> > > On Sat, Mar 26, 2011 at 07:05:44AM +0800, Jan Kara wrote:
> > > > And actually the NFS traces you pointed to originally seem to be different
> > > > problem, in fact not directly related to what balance_dirty_pages() does...
> > > > And with local filesystem the results seem to be reasonable (although there
> > > > are some longer sleeps in your JBOD measurements I don't understand yet).
> > > 
> > > Yeah the NFS case can be improved on the FS side (for now you may just
> > > reuse my NFS patches and focus on other generic improvements).
> > > 
> > > The JBOD issue is also beyond my understanding.
> > > 
> > > Note that XFS will also see one big IO completion per 0.5-1 seconds,
> > > when we are to increase the write chunk size from the current 4MB to
> > > near the bdi's write bandwidth. As illustrated by this graph:
> > > 
> > > http://www.kernel.org/pub/linux/kernel/people/wfg/writeback/dirty-throttling-v6/4G/xfs-1dd-1M-8p-3927M-20%25-2.6.38-rc6-dt6+-2011-02-27-22-58/global_dirtied_written-500.png
> > 
> > Which is _bad_.
> > 
> > Increasing the writeback chunk size simply causes dirty queue
> > starvation issues when there are lots of dirty files and lots more
> > memory than there is writeback bandwidth. Think of a machine with
> > 1TB of RAM (that's a 200GB dirty limit) and 1GB/s of disk
> > throughput. Thats 3 minutes worth of writeback and increasing the
> > chunk size to ~1s worth of throughput means that the 200th dirty
> > file won't get serviced for 3 minutes....
> > 
> > We used to have behaviour similar to this this (prior to 2.6.16, IIRC),
> > and it caused all sorts of problems where people were losing 10-15
> > minute old data when the system crashed because writeback didn't
> > process the dirty inode list fast enough in the presence of lots of
> > large files....
>  
> Yes it is a problem, and can be best solved by automatically lowering
> bdi dirty limit to (bdi->write_bandwidth * dirty_expire_interval/100).
> Then we reliably control the lost data size to < 30s by default.

Perhaps, though I see problems with that also. e.g. write bandwidth
is 100MB/s (dirty limit ~= 3GB), then someone runs a find on the
same disk and write bandwidth drops to 10MB/s (dirty limit changes
to ~300MB). Then we are 10x over the new dirty limit and the
writing application will be completely throttled for the next 270s
until the dirty pages drop below the new dirty limit or the find
stops.

IOWs, it changing IO workloads will cause interesting corner cases
to be discovered and hence further complexity to handle effectively.
And trying to diagnose problems because of such changes in IO load
will be nigh on impossible - how would you gather sufficient
information to determine that application A stalled for a minute
because application B read a bunch of stuff from disk at the wrong
time? Then how would you prove that you'd fixed the problem without
introducing some other regression triggered by different workload
changes?

> > A small writeback chunk size has no adverse impact on XFS as long as
> > the elevator does it's job of merging IOs (which in 99.9% of cases
> > it does) so I'm wondering what the reason for making this change
> > is.
> 
> It's explained in this changelog (is the XFS paragraph still valid?)
> 
>         https://patchwork.kernel.org/patch/605151/

You mean this paragraph?

"According to Christoph, the current writeback size is way too
small, and XFS had a hack that bumped out nr_to_write to four times
the value sent by the VM to be able to saturate medium-sized RAID
arrays.  This value was also problematic for ext4 as well, as it
caused large files to be come interleaved on disk by in 8 megabyte
chunks (we bumped up the nr_to_write by a factor of two)."

We _used_ to have such a hack. It was there from 2.6.30 through to
2.6.35 - from when we realised writeback had bitrotted into badness
to when we fixed the last set of bugs that the nr_to_write windup
was papering over. between 2.6.30 and 2.6.35 we changed to dedicated
flusher threads, got rid of congestion backoff, fixed up a bunch
of queueing issues and finally stopped nr_to_write from going and
staying negative and getting stuck on single inodes until they had
no more dirty pages left. That was when this was committed:

commit 254c8c2dbf0e06a560a5814eb90cb628adb2de66
Author: Dave Chinner <dchinner@xxxxxxxxxx>
Date:   Wed Jun 9 10:37:19 2010 +1000

    xfs: remove nr_to_write writeback windup.
    
    Now that the background flush code has been fixed, we shouldn't need to
    silently multiply the wbc->nr_to_write to get good writeback. Remove
    that code.
    
    Signed-off-by: Dave Chinner <dchinner@xxxxxxxxxx>
    Reviewed-by: Christoph Hellwig <hch@xxxxxx>
    Signed-off-by: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>

And writeback throughput is now as good as it ever was....

> The larger write chunk size generally helps ext4 and RAID setups.

Is this still true with ext4's new submit_bio()-based writeback IO
submission path that was copied from the XFS? It's a lot more
efficient so should be much better on RAID setups.

Cheers,

Dave.
-- 
Dave Chinner
david@xxxxxxxxxxxxx

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxxx  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]