Re: [patch 0/5] mm: per-zone dirty limiting

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Jul 29, 2011 at 12:05:10PM +0100, Mel Gorman wrote:
> On Tue, Jul 26, 2011 at 08:05:59PM +0200, Johannes Weiner wrote:
> > > As dd is variable, I'm rerunning the tests to do 4 iterations and
> > > multiple memory sizes for just xfs and ext4 to see what falls out. It
> > > should take about 14 hours to complete assuming nothing screws up.
> > 
> > Awesome, thanks!
> > 
> 
> While they in fact took about 30 hours to complete, I only got around
> to packaging them up now. Unfortuantely the tests were incomplete as
> I needed the machine back for another use but the results that did
> complete are at http://www.csn.ul.ie/~mel/postings/hnaz-20110729/
> 
> Look for the comparison.html files such as this one
> 
> http://www.csn.ul.ie/~mel/postings/hnaz-20110729/global-dhp-512M__writeback-reclaimdirty-ext3/hydra/comparison.html
> 
> I'm afraid I haven't looked through them in detail.

Mel, thanks a lot for running those tests, you shall be compensated in
finest brewery goods some time.

Here is an attempt:

	global-dhp-512M__writeback-reclaimdirty-xfs

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0      3.0.0-lessks
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1      pzdirty-v3r1
1                    1054.54 ( 0.00%) 386.65 (172.74%) 375.60 (180.76%) 375.88 (180.55%)
                 +/-            1.41%            4.56%            3.09%            2.34%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         32.27     29.97     30.65     30.91
Total Elapsed Time (seconds)               4220.48   1548.84   1504.64   1505.79

MMTests Statistics: vmstat
Page Ins                                    720433    392017    317097    343849
Page Outs                                 27746435  27673017  27619134  27555437
Swap Ins                                    173563     94196     74844     81954
Swap Outs                                   115864    100264     86833     70904
Direct pages scanned                       3268014      7515         0      1008
Kswapd pages scanned                       5351371  12045948   7973273   7923387
Kswapd pages reclaimed                     3320848   6498700   6486754   6492607
Direct pages reclaimed                     3267145      7243         0      1008
Kswapd efficiency                              62%       53%       81%       81%
Kswapd velocity                           1267.953  7777.400  5299.123  5261.947
Direct efficiency                              99%       96%      100%      100%
Direct velocity                            774.323     4.852     0.000     0.669
Percentage direct scans                        37%        0%        0%        0%
Page writes by reclaim                      130541    100265     86833     70904
Page writes file                             14677         1         0         0
Page writes anon                            115864    100264     86833     70904
Page reclaim invalidate                          0   3120195         0         0
Slabs scanned                                 8448      8448      8576      8448
Direct inode steals                              0         0         0         0
Kswapd inode steals                           1828      1837      2056      1918
Kswapd skipped wait                              0         1         0         0
Compaction stalls                                2         0         0         0
Compaction success                               1         0         0         0
Compaction failures                              1         0         0         0
Compaction pages moved                           0         0         0         0
Compaction move failure                          0         0         0         0

While file writes from reclaim are prevented by both patches on their
own, perzonedirty decreases the amount of anonymous pages swapped out
because reclaim is always able to make progress instead of wasting its
file scan budget on shuffling dirty pages.  With lesskswapd in
addition, swapping is throttled in reclaim by the ratio of dirty pages
to isolated pages.

The runtime improvements speak for both perzonedirty and
perzonedirty+lesskswapd.  Given the swap upside and increased reclaim
efficiency, the combination of both appears to be the most desirable.

	global-dhp-512M__writeback-reclaimdirty-ext3

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    1762.23 ( 0.00%) 987.73 (78.41%) 983.82 (79.12%)
                 +/-            4.35%            2.24%            1.56%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         46.36     44.07        46
Total Elapsed Time (seconds)               7053.28   3956.60   3940.39

MMTests Statistics: vmstat
Page Ins                                    965236    661660    629972
Page Outs                                 27984332  27922904  27715628
Swap Ins                                    231181    158799    137341
Swap Outs                                   151395    142150     88644
Direct pages scanned                       2749884     11138   1315072
Kswapd pages scanned                       6340921  12591169   6599999
Kswapd pages reclaimed                     3915635   6576549   5264406
Direct pages reclaimed                     2749002     10877   1314842
Kswapd efficiency                              61%       52%       79%
Kswapd velocity                            899.003  3182.320  1674.961
Direct efficiency                              99%       97%       99%
Direct velocity                            389.873     2.815   333.742
Percentage direct scans                        30%        0%       16%
Page writes by reclaim                      620698    142155     88645
Page writes file                            469303         5         1
Page writes anon                            151395    142150     88644
Page reclaim invalidate                          0   3717819         0
Slabs scanned                                 8704      8576     33408
Direct inode steals                              0         0       466
Kswapd inode steals                           1872      2107      2115
Kswapd skipped wait                              0         1         0
Compaction stalls                                2         0         1
Compaction success                               1         0         0
Compaction failures                              1         0         1
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

perzonedirty the highest reclaim efficiencies, the lowest writeout
counts from reclaim, and the shortest runtime.

While file writes are practically gone with both lesskswapd and
perzonedirty on their own, the latter also reduces swapping by 40%.

I expect the combination of both series to have the best results here
as well.

	global-dhp-512M__writeback-reclaimdirty-ext4

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    405.42 ( 0.00%) 410.48 (-1.23%) 401.77 ( 0.91%)
                 +/-            3.62%            4.45%            2.82%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         31.25      31.4     31.37
Total Elapsed Time (seconds)               1624.60   1644.56   1609.67

MMTests Statistics: vmstat
Page Ins                                    354364    403612    332812
Page Outs                                 27607792  27709096  27536412
Swap Ins                                     84065     96398     79219
Swap Outs                                    83096    108478     65342
Direct pages scanned                           112         0        56
Kswapd pages scanned                      12207898  12063862   7615377
Kswapd pages reclaimed                     6492490   6504947   6486946
Direct pages reclaimed                         112         0        56
Kswapd efficiency                              53%       53%       85%
Kswapd velocity                           7514.402  7335.617  4731.018
Direct efficiency                             100%      100%      100%
Direct velocity                              0.069     0.000     0.035
Percentage direct scans                         0%        0%        0%
Page writes by reclaim                     3076760    108483     65342
Page writes file                           2993664         5         0
Page writes anon                             83096    108478     65342
Page reclaim invalidate                          0   3291697         0
Slabs scanned                                 8448      8448      8448
Direct inode steals                              0         0         0
Kswapd inode steals                           1979      1993      1945
Kswapd skipped wait                              1         0         0
Compaction stalls                                0         0         0
Compaction success                               0         0         0
Compaction failures                              0         0         0
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

With lesskswapd, both runtime and swapouts increased.  My only guess
is that in this configuration, the writepage calls actually improve
things to a certain extent.

Otherwise, nothing stands out to me here, and the same as above
applies wrt runtime and reclaim efficiency being the best with
perzonedirty.

	global-dhp-1024M__writeback-reclaimdirty-ext3

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    1291.74 ( 0.00%) 1034.56 (24.86%) 1023.04 (26.26%)
                 +/-            2.77%            1.98%            4.42%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         42.41     41.97     43.49
Total Elapsed Time (seconds)               5176.73   4142.26   4096.57

MMTests Statistics: vmstat
Page Ins                                     27856     24392     23292
Page Outs                                 27360416  27352736  27352700
Swap Ins                                         1         6         0
Swap Outs                                        2        39        32
Direct pages scanned                          5899         0         0
Kswapd pages scanned                       6500396   7948564   6014854
Kswapd pages reclaimed                     6008477   6012586   6013794
Direct pages reclaimed                        5899         0         0
Kswapd efficiency                              92%       75%       99%
Kswapd velocity                           1255.695  1918.895  1468.266
Direct efficiency                             100%      100%      100%
Direct velocity                              1.140     0.000     0.000
Percentage direct scans                         0%        0%        0%
Page writes by reclaim                      181091        39        32
Page writes file                            181089         0         0
Page writes anon                                 2        39        32
Page reclaim invalidate                          0   1843189         0
Slabs scanned                                 3840      3840      4096
Direct inode steals                              0         0         0
Kswapd inode steals                              0         0         0
Kswapd skipped wait                              0         0         0
Compaction stalls                                0         0         0
Compaction success                               0         0         0
Compaction failures                              0         0         0
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

Writes from reclaim are reduced to practically nothing by both
patchsets, but perzonedirty standalone wins in runtime and reclaim
efficiency.

	global-dhp-1024M__writeback-reclaimdirty-ext4

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    434.46 ( 0.00%) 432.42 ( 0.47%) 429.15 ( 1.24%)
                 +/-            2.62%            2.15%            2.47%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         29.44     29.37     29.64
Total Elapsed Time (seconds)               1740.46   1732.34   1719.08

MMTests Statistics: vmstat
Page Ins                                     15216     14728     12936
Page Outs                                 27274352  27274144  27274236
Swap Ins                                        12         0         7
Swap Outs                                       13         0        29
Direct pages scanned                             0         0         0
Kswapd pages scanned                       8151970   7662106   5989819
Kswapd pages reclaimed                     5990667   5987919   5988646
Direct pages reclaimed                           0         0         0
Kswapd efficiency                              73%       78%       99%
Kswapd velocity                           4683.802  4422.980  3484.317
Direct efficiency                             100%      100%      100%
Direct velocity                              0.000     0.000     0.000
Percentage direct scans                         0%        0%        0%
Page writes by reclaim                     1889005         0        29
Page writes file                           1888992         0         0
Page writes anon                                13         0        29
Page reclaim invalidate                          0   1574594         0
Slabs scanned                                 3968      3840      3968
Direct inode steals                              0         0         0
Kswapd inode steals                              0         0         0
Kswapd skipped wait                              0         0         0
Compaction stalls                                0         0         0
Compaction success                               0         0         0
Compaction failures                              0         0         0
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

As with ext3, perzonedirty is best in overall runtime and reclaim
efficiency.

	global-dhp-1024M__writeback-reclaimdirty-xfs

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    757.46 ( 0.00%) 387.51 (95.47%) 381.90 (98.34%)
                 +/-            3.03%            1.41%            1.13%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         28.68     27.86     29.25
Total Elapsed Time (seconds)               3032.05   1552.22   1529.82

MMTests Statistics: vmstat
Page Ins                                     23325     13801     13733
Page Outs                                 27277838  27271665  27272055
Swap Ins                                         1         0         0
Swap Outs                                       24         0        58
Direct pages scanned                         37729         0         0
Kswapd pages scanned                       6340969   7643093   5994387
Kswapd pages reclaimed                     5959043   5990117   5993349
Direct pages reclaimed                       37388         0         0
Kswapd efficiency                              93%       78%       99%
Kswapd velocity                           2091.314  4923.975  3918.361
Direct efficiency                              99%      100%      100%
Direct velocity                             12.443     0.000     0.000
Percentage direct scans                         0%        0%        0%
Page writes by reclaim                        7148         0        58
Page writes file                              7124         0         0
Page writes anon                                24         0        58
Page reclaim invalidate                          0   1552818         0
Slabs scanned                                 4224      3968      3968
Direct inode steals                              0         0         0
Kswapd inode steals                              0         0         0
Kswapd skipped wait                              0         0         0
Compaction stalls                                0         0         0
Compaction success                               0         0         0
Compaction failures                              0         0         0
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

As with ext3 and ext4, perzonedirty is best in overall runtime and
reclaim efficiency.

	global-dhp-4608M__writeback-reclaimdirty-ext3

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    1274.37 ( 0.00%) 1204.00 ( 5.84%) 1317.79 (-3.29%)
                 +/-            2.02%            2.03%            3.05%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         43.93      44.4     45.85
Total Elapsed Time (seconds)               5130.22   4824.17   5278.84

MMTests Statistics: vmstat
Page Ins                                     44004     43704     44492
Page Outs                                 27391592  27386240  27390108
Swap Ins                                      6968      5855      6091
Swap Outs                                     8846      8024      8065
Direct pages scanned                             0         0    115384
Kswapd pages scanned                       4234168   4656846   4105795
Kswapd pages reclaimed                     3899101   3893500   3776056
Direct pages reclaimed                           0         0    115347
Kswapd efficiency                              92%       83%       91%
Kswapd velocity                            825.338   965.315   777.784
Direct efficiency                             100%      100%       99%
Direct velocity                              0.000     0.000    21.858
Percentage direct scans                         0%        0%        2%
Page writes by reclaim                       42555      8024     40622
Page writes file                             33709         0     32557
Page writes anon                              8846      8024      8065
Page reclaim invalidate                          0    586463         0
Slabs scanned                                 3712      3840      3840
Direct inode steals                              0         0         0
Kswapd inode steals                              0         0         0
Kswapd skipped wait                              0         0         0
Compaction stalls                                0         0         0
Compaction success                               0         0         0
Compaction failures                              0         0         0
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

Here, perzonedirty fails to ensure enough clean pages in what I guess
is a small Normal zone on top of the DMA32 zone.  The
(not-yet-optimized) per-zone dirty checks cost CPU time but they do
not pay off and dirty pages are still encountered by reclaim.

Mel, can you say how big exactly the Normal zone is with this setup?

My theory is that the closer (file_pages - dirty_pages) is to the high
watermark which kswapd tries to balance to, the more likely it is to
run into dirty pages.  And to my knowledge, these tests are run with a
non-standard 40% dirty ratio, which lowers the threshold at which
perzonedirty falls apart.  Per-zone dirty limits should probably take
the high watermark into account.

This does not explain the regression to me, however, if the Normal
zone here is about the same size as the DMA32 zone in the 512M tests
above, for which perzonedirty was an unambiguous improvement.

What makes me wonder, is that in addition, something in perzonedirty
makes kswapd less efficient in the 4G tests, which is the opposite
effect it had in all other setups.  This increases direct reclaim
invocations against the preferred Normal zone.  The higher pressure
could also explain why reclaim rushes through the clean pages and runs
into dirty pages quicker.

Does anyone have a theory about what might be going on here?

The tests with other filesystems on 4G memory look similarly bleak for
perzonedirty:

	global-dhp-4608M__writeback-reclaimdirty-ext4

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    396.85 ( 0.00%) 437.61 (-9.31%) 404.65 (-1.93%)
                 +/-            13.10%            16.04%            16.35%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         30.46     30.52     32.28
Total Elapsed Time (seconds)               1591.42   1754.49   1622.63

MMTests Statistics: vmstat
Page Ins                                     37316     38984     36816
Page Outs                                 27304668  27305952  27307584
Swap Ins                                      6705      6728      6840
Swap Outs                                     7989      7911      8431
Direct pages scanned                             0         0         0
Kswapd pages scanned                       4627064   4644718   4618129
Kswapd pages reclaimed                     3883654   3891597   3878173
Direct pages reclaimed                           0         0         0
Kswapd efficiency                              83%       83%       83%
Kswapd velocity                           2907.507  2647.332  2846.076
Direct efficiency                             100%      100%      100%
Direct velocity                              0.000     0.000     0.000
Percentage direct scans                         0%        0%        0%
Page writes by reclaim                      586753      7911    588292
Page writes file                            578764         0    579861
Page writes anon                              7989      7911      8431
Page reclaim invalidate                          0    591028         0
Slabs scanned                                 3840      3840      4096
Direct inode steals                              0         0         0
Kswapd inode steals                              0         0         0
Kswapd skipped wait                              0         0         0
Compaction stalls                                0         0         0
Compaction success                               0         0         0
Compaction failures                              0         0         0
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

	global-dhp-4608M__writeback-reclaimdirty-xfs

SIMPLE WRITEBACK
              simple-writeback   writeback-3.0.0   writeback-3.0.0
                 3.0.0-vanilla   lesskswapd-v3r1 perzonedirty-v1r1
1                    531.54 ( 0.00%) 404.88 (31.28%) 546.32 (-2.71%)
                 +/-            1.77%            7.06%            1.01%
MMTests Statistics: duration
User/Sys Time Running Test (seconds)         29.35     30.04     30.63
Total Elapsed Time (seconds)               2129.69   1623.11   2188.73

MMTests Statistics: vmstat
Page Ins                                     38329     37173     35117
Page Outs                                 27307040  27304636  27305927
Swap Ins                                      6469      6239      5138
Swap Outs                                     8292      8299      7934
Direct pages scanned                             0         0    117901
Kswapd pages scanned                       4197481   4630492   4060306
Kswapd pages reclaimed                     3880444   3882479   3767544
Direct pages reclaimed                           0         0    117872
Kswapd efficiency                              92%       83%       92%
Kswapd velocity                           1970.935  2852.852  1855.097
Direct efficiency                             100%      100%       99%
Direct velocity                              0.000     0.000    53.867
Percentage direct scans                         0%        0%        2%
Page writes by reclaim                        9667      8299      9249
Page writes file                              1375         0      1315
Page writes anon                              8292      8299      7934
Page reclaim invalidate                          0    575703         0
Slabs scanned                                 3840      3712      4352
Direct inode steals                              0         0         0
Kswapd inode steals                              0         0         0
Kswapd skipped wait                              0         0         0
Compaction stalls                                0         0         0
Compaction success                               0         0         0
Compaction failures                              0         0         0
Compaction pages moved                           0         0         0
Compaction move failure                          0         0         0

I am doubly confused because I ran similar tests with 4G memory and
got contradicting results.  Will rerun those to make sure.

Comments?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]