On Sun, Oct 09, 2011 at 08:27:36AM +0800, Wu Fengguang wrote:
> On Sat, Oct 08, 2011 at 09:49:27PM +0800, Wu Fengguang wrote:
> > On Sat, Oct 08, 2011 at 07:52:27PM +0800, Wu Fengguang wrote:
> > > On Sat, Oct 08, 2011 at 12:00:36PM +0800, Wu Fengguang wrote:
> > > > Hi Jan,
> > > >
> > > > The test results look not good: btrfs is heavily impacted and the
> > > > other filesystems are slightly impacted.
> > > >
> > > > I'll send you the detailed logs in private emails (too large for the
> > > > mailing list). Basically I noticed many writeback_wait traces that
> > > > never appear w/o this patch. In the btrfs cases that see larger
> > > > regressions, I see large fluctuations in the writeout bandwidth and
> > > > long disk idle periods. It's still a bit puzzling how all these
> > > > happen..
> > >
> > > Sorry I find that part of the regressions (about 2-3%) are caused by
> > > change of my test scripts recently. Here are the more fair compares
> > > and they show only regressions in btrfs and xfs:
> > >
> > > 3.1.0-rc8-ioless6a+ 3.1.0-rc8-ioless6-requeue+
> > > ------------------------ ------------------------
> > > 37.34 +0.8% 37.65 thresh=100M/ext3-10dd-4k-8p-4096M-100M:10-X
> > > 44.44 +3.4% 45.96 thresh=100M/ext3-1dd-4k-8p-4096M-100M:10-X
> > > 41.70 +1.0% 42.14 thresh=100M/ext3-2dd-4k-8p-4096M-100M:10-X
> > > 46.45 -0.3% 46.32 thresh=100M/ext4-10dd-4k-8p-4096M-100M:10-X
> > > 56.60 -0.3% 56.41 thresh=100M/ext4-1dd-4k-8p-4096M-100M:10-X
> > > 54.14 +0.9% 54.63 thresh=100M/ext4-2dd-4k-8p-4096M-100M:10-X
> > > 30.66 -0.7% 30.44 thresh=1G/ext3-100dd-4k-8p-4096M-1024M:10-X
> > > 35.24 +1.6% 35.82 thresh=1G/ext3-10dd-4k-8p-4096M-1024M:10-X
> > > 43.58 +0.5% 43.80 thresh=1G/ext3-1dd-4k-8p-4096M-1024M:10-X
> > > 50.42 -0.6% 50.14 thresh=1G/ext4-100dd-4k-8p-4096M-1024M:10-X
> > > 56.23 -1.0% 55.64 thresh=1G/ext4-10dd-4k-8p-4096M-1024M:10-X
> > > 58.12 -0.5% 57.84 thresh=1G/ext4-1dd-4k-8p-4096M-1024M:10-X
> > > 45.37 +1.4% 46.03 thresh=8M/ext3-1dd-4k-8p-4096M-8M:10-X
> > > 43.71 +2.2% 44.69 thresh=8M/ext3-2dd-4k-8p-4096M-8M:10-X
> > > 35.58 +0.5% 35.77 thresh=8M/ext4-10dd-4k-8p-4096M-8M:10-X
> > > 56.39 +1.4% 57.16 thresh=8M/ext4-1dd-4k-8p-4096M-8M:10-X
> > > 51.26 +1.5% 52.04 thresh=8M/ext4-2dd-4k-8p-4096M-8M:10-X
> > > 787.25 +0.7% 792.47 TOTAL
> > >
> > > 3.1.0-rc8-ioless6a+ 3.1.0-rc8-ioless6-requeue+
> > > ------------------------ ------------------------
> > > 44.53 -18.6% 36.23 thresh=100M/xfs-10dd-4k-8p-4096M-100M:10-X
> > > 55.89 -0.4% 55.64 thresh=100M/xfs-1dd-4k-8p-4096M-100M:10-X
> > > 51.11 +0.5% 51.35 thresh=100M/xfs-2dd-4k-8p-4096M-100M:10-X
> > > 41.76 -4.8% 39.77 thresh=1G/xfs-100dd-4k-8p-4096M-1024M:10-X
> > > 48.34 -0.3% 48.18 thresh=1G/xfs-10dd-4k-8p-4096M-1024M:10-X
> > > 52.36 -0.2% 52.26 thresh=1G/xfs-1dd-4k-8p-4096M-1024M:10-X
> > > 31.07 -1.1% 30.74 thresh=8M/xfs-10dd-4k-8p-4096M-8M:10-X
> > > 55.44 -0.6% 55.09 thresh=8M/xfs-1dd-4k-8p-4096M-8M:10-X
> > > 47.59 -31.2% 32.74 thresh=8M/xfs-2dd-4k-8p-4096M-8M:10-X
> > > 428.07 -6.1% 401.99 TOTAL
> > >
> > > 3.1.0-rc8-ioless6a+ 3.1.0-rc8-ioless6-requeue+
> > > ------------------------ ------------------------
> > > 58.23 -82.6% 10.13 thresh=100M/btrfs-10dd-4k-8p-4096M-100M:10-X
> > > 58.43 -80.3% 11.54 thresh=100M/btrfs-1dd-4k-8p-4096M-100M:10-X
> > > 58.53 -79.9% 11.76 thresh=100M/btrfs-2dd-4k-8p-4096M-100M:10-X
> > > 56.55 -31.7% 38.63 thresh=1G/btrfs-100dd-4k-8p-4096M-1024M:10-X
> > > 56.11 -30.1% 39.25 thresh=1G/btrfs-10dd-4k-8p-4096M-1024M:10-X
> > > 56.21 -18.3% 45.93 thresh=1G/btrfs-1dd-4k-8p-4096M-1024M:10-X
> > > 344.06 -54.3% 157.24 TOTAL
> > >
> > > I'm now bisecting the patches to find out the root cause.
> >
> > Current findings are, when only applying the first patch, or reduce the second
> > patch to the below one, the btrfs regressions are restored:
>
> And the below reduced patch is also OK:
>
> 3.1.0-rc8-ioless6a+ 3.1.0-rc8-ioless6-requeue4+
> ------------------------ ------------------------
> 58.23 -0.4% 57.98 thresh=100M/btrfs-10dd-4k-8p-4096M-100M:10-X
> 58.43 -2.2% 57.13 thresh=100M/btrfs-1dd-4k-8p-4096M-100M:10-X
> 58.53 -1.2% 57.83 thresh=100M/btrfs-2dd-4k-8p-4096M-100M:10-X
> 37.34 -0.7% 37.07 thresh=100M/ext3-10dd-4k-8p-4096M-100M:10-X
> 44.44 +0.2% 44.52 thresh=100M/ext3-1dd-4k-8p-4096M-100M:10-X
> 41.70 +0.0% 41.72 thresh=100M/ext3-2dd-4k-8p-4096M-100M:10-X
> 46.45 -0.7% 46.10 thresh=100M/ext4-10dd-4k-8p-4096M-100M:10-X
> 56.60 -0.8% 56.15 thresh=100M/ext4-1dd-4k-8p-4096M-100M:10-X
> 54.14 +0.3% 54.33 thresh=100M/ext4-2dd-4k-8p-4096M-100M:10-X
> 44.53 -7.3% 41.29 thresh=100M/xfs-10dd-4k-8p-4096M-100M:10-X
> 55.89 +0.9% 56.39 thresh=100M/xfs-1dd-4k-8p-4096M-100M:10-X
> 51.11 +1.0% 51.60 thresh=100M/xfs-2dd-4k-8p-4096M-100M:10-X
> 56.55 -1.0% 55.97 thresh=1G/btrfs-100dd-4k-8p-4096M-1024M:10-X
> 56.11 -1.5% 55.28 thresh=1G/btrfs-10dd-4k-8p-4096M-1024M:10-X
> 56.21 -1.9% 55.16 thresh=1G/btrfs-1dd-4k-8p-4096M-1024M:10-X
> 30.66 -2.7% 29.82 thresh=1G/ext3-100dd-4k-8p-4096M-1024M:10-X
> 35.24 -0.7% 35.00 thresh=1G/ext3-10dd-4k-8p-4096M-1024M:10-X
> 43.58 -2.1% 42.65 thresh=1G/ext3-1dd-4k-8p-4096M-1024M:10-X
> 50.42 -2.4% 49.21 thresh=1G/ext4-100dd-4k-8p-4096M-1024M:10-X
> 56.23 -2.2% 55.00 thresh=1G/ext4-10dd-4k-8p-4096M-1024M:10-X
> 58.12 -1.8% 57.08 thresh=1G/ext4-1dd-4k-8p-4096M-1024M:10-X
> 41.76 -5.1% 39.61 thresh=1G/xfs-100dd-4k-8p-4096M-1024M:10-X
> 48.34 -2.6% 47.06 thresh=1G/xfs-10dd-4k-8p-4096M-1024M:10-X
> 52.36 -3.3% 50.64 thresh=1G/xfs-1dd-4k-8p-4096M-1024M:10-X
> 45.37 +0.7% 45.70 thresh=8M/ext3-1dd-4k-8p-4096M-8M:10-X
> 43.71 +0.7% 44.00 thresh=8M/ext3-2dd-4k-8p-4096M-8M:10-X
> 35.58 +0.7% 35.82 thresh=8M/ext4-10dd-4k-8p-4096M-8M:10-X
> 56.39 -1.1% 55.77 thresh=8M/ext4-1dd-4k-8p-4096M-8M:10-X
> 51.26 -0.6% 50.94 thresh=8M/ext4-2dd-4k-8p-4096M-8M:10-X
> 31.07 -13.3% 26.94 thresh=8M/xfs-10dd-4k-8p-4096M-8M:10-X
> 55.44 +0.5% 55.72 thresh=8M/xfs-1dd-4k-8p-4096M-8M:10-X
> 47.59 +1.6% 48.33 thresh=8M/xfs-2dd-4k-8p-4096M-8M:10-X
> 1559.39 -1.4% 1537.83 TOTAL
Now I got slightly better results with the below incremental patch.
3.1.0-rc8-ioless6a+ 3.1.0-rc8-ioless6-requeue5+
------------------------ ------------------------
58.23 +0.9% 58.76 thresh=100M/btrfs-10dd-4k-8p-4096M-100M:10-X
58.43 -0.4% 58.19 thresh=100M/btrfs-1dd-4k-8p-4096M-100M:10-X
58.53 -0.1% 58.48 thresh=100M/btrfs-2dd-4k-8p-4096M-100M:10-X
37.34 +1.4% 37.88 thresh=100M/ext3-10dd-4k-8p-4096M-100M:10-X
44.44 -0.3% 44.30 thresh=100M/ext3-1dd-4k-8p-4096M-100M:10-X
41.70 +1.3% 42.24 thresh=100M/ext3-2dd-4k-8p-4096M-100M:10-X
46.45 +0.3% 46.59 thresh=100M/ext4-10dd-4k-8p-4096M-100M:10-X
56.60 -0.7% 56.22 thresh=100M/ext4-1dd-4k-8p-4096M-100M:10-X
54.14 +0.7% 54.50 thresh=100M/ext4-2dd-4k-8p-4096M-100M:10-X
44.53 -2.4% 43.45 thresh=100M/xfs-10dd-4k-8p-4096M-100M:10-X
55.89 +0.2% 56.00 thresh=100M/xfs-1dd-4k-8p-4096M-100M:10-X
51.11 -0.0% 51.10 thresh=100M/xfs-2dd-4k-8p-4096M-100M:10-X
56.55 +0.0% 56.56 thresh=1G/btrfs-100dd-4k-8p-4096M-1024M:10-X
56.11 +0.5% 56.38 thresh=1G/btrfs-10dd-4k-8p-4096M-1024M:10-X
56.21 +0.1% 56.29 thresh=1G/btrfs-1dd-4k-8p-4096M-1024M:10-X
30.66 +0.2% 30.72 thresh=1G/ext3-100dd-4k-8p-4096M-1024M:10-X
35.24 +0.8% 35.54 thresh=1G/ext3-10dd-4k-8p-4096M-1024M:10-X
43.58 +0.1% 43.64 thresh=1G/ext3-1dd-4k-8p-4096M-1024M:10-X
50.42 -0.9% 49.99 thresh=1G/ext4-100dd-4k-8p-4096M-1024M:10-X
56.23 -0.0% 56.21 thresh=1G/ext4-10dd-4k-8p-4096M-1024M:10-X
58.12 -1.9% 57.02 thresh=1G/ext4-1dd-4k-8p-4096M-1024M:10-X
41.76 -5.9% 39.30 thresh=1G/xfs-100dd-4k-8p-4096M-1024M:10-X
48.34 -1.4% 47.67 thresh=1G/xfs-10dd-4k-8p-4096M-1024M:10-X
52.36 -1.8% 51.41 thresh=1G/xfs-1dd-4k-8p-4096M-1024M:10-X
54.37 +0.6% 54.71 thresh=8M/btrfs-10dd-4k-8p-4096M-8M:10-X
56.11 +1.7% 57.08 thresh=8M/btrfs-1dd-4k-8p-4096M-8M:10-X
56.22 +0.8% 56.67 thresh=8M/btrfs-2dd-4k-8p-4096M-8M:10-X
32.21 -0.2% 32.15 thresh=8M/ext3-10dd-4k-8p-4096M-8M:10-X
45.37 +0.2% 45.45 thresh=8M/ext3-1dd-4k-8p-4096M-8M:10-X
43.71 +0.0% 43.72 thresh=8M/ext3-2dd-4k-8p-4096M-8M:10-X
35.58 +0.1% 35.61 thresh=8M/ext4-10dd-4k-8p-4096M-8M:10-X
56.39 +1.0% 56.98 thresh=8M/ext4-1dd-4k-8p-4096M-8M:10-X
51.26 +0.4% 51.44 thresh=8M/ext4-2dd-4k-8p-4096M-8M:10-X
31.07 -11.5% 27.49 thresh=8M/xfs-10dd-4k-8p-4096M-8M:10-X
55.44 +0.7% 55.81 thresh=8M/xfs-1dd-4k-8p-4096M-8M:10-X
47.59 +1.0% 48.05 thresh=8M/xfs-2dd-4k-8p-4096M-8M:10-X
1758.29 -0.3% 1753.58 TOTAL
3.1.0-rc8-ioless6a+ 3.1.0-rc8-ioless6-requeue5+
------------------------ ------------------------
62.64 -0.8% 62.13 3G-UKEY-HDD/ext4-1dd-4k-8p-4096M-20:10-X
59.80 -0.1% 59.76 3G-UKEY-HDD/ext4-2dd-4k-8p-4096M-20:10-X
53.42 -1.1% 52.84 3G-UKEY-HDD/xfs-10dd-4k-8p-4096M-20:10-X
59.00 -0.1% 58.92 3G-UKEY-HDD/xfs-1dd-4k-8p-4096M-20:10-X
56.14 -0.3% 55.95 3G-UKEY-HDD/xfs-2dd-4k-8p-4096M-20:10-X
291.00 -0.5% 289.60 TOTAL
> @@ -597,10 +610,10 @@ static long writeback_sb_inodes(struct s
> wrote++;
> if (wbc.pages_skipped) {
> /*
> - * writeback is not making progress due to locked
> - * buffers. Skip this inode for now.
> + * Writeback is not making progress due to unavailable
> + * fs locks or similar condition. Retry in next round.
> */
> - redirty_tail(inode, wb);
> + requeue_io(inode, wb);
> }
> spin_unlock(&inode->i_lock);
> spin_unlock(&wb->list_lock);
The above change to requeue_io() is not desirable in the case the
inode was redirty_tail()ed in writeback_singel_inode(). So I'd propose
to just leave the original code as it is, or to remove the code as the
below patch. In either way, we get slightly better performance
demonstrated by the larger bandwidth in 3.1.0-rc8-ioless6-requeue3+
and 3.1.0-rc8-ioless6-requeue5+.
Thanks,
Fengguang
---
--- linux-next.orig/fs/fs-writeback.c 2011-10-09 08:34:38.000000000 +0800
+++ linux-next/fs/fs-writeback.c 2011-10-09 08:35:06.000000000 +0800
@@ -608,13 +608,6 @@ static long writeback_sb_inodes(struct s
wrote += write_chunk - wbc.nr_to_write;
if (!(inode->i_state & I_DIRTY))
wrote++;
- if (wbc.pages_skipped) {
- /*
- * Writeback is not making progress due to unavailable
- * fs locks or similar condition. Retry in next round.
- */
- requeue_io(inode, wb);
- }
spin_unlock(&inode->i_lock);
spin_unlock(&wb->list_lock);
iput(inode);
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
