On Wed, Jun 25, 2014 at 04:32:50PM -0700, Andrew Morton wrote: > On Wed, 25 Jun 2014 08:58:46 +0100 Mel Gorman <mgorman@xxxxxxx> wrote: > > > Historically kswapd scanned from DMA->Movable in the opposite direction > > to the page allocator to avoid allocating behind kswapd direction of > > progress. The fair zone allocation policy altered this in a non-obvious > > manner. > > > > Traditionally, the page allocator prefers to use the highest eligible zone > > until the watermark is depleted, woke kswapd and moved onto the next zone. > > kswapd scans zones in the opposite direction so the scanning lists on > > 64-bit look like this; > > > > ... > > > > Note that this patch makes a large performance difference for lower > > numbers of threads and brings performance closer to 3.0 figures. It was > > also tested against xfs and there are similar gains although I don't have > > 3.0 figures to compare against. There are still regressions for higher > > number of threads but this is related to changes in the CFQ IO scheduler. > > > > Why did this patch make a difference to sequential read performance? > IOW, by what means was/is reclaim interfering with sequential reads? > The fair zone allocator is interleaving between Normal/DMA32. Kswapd is reclaiming from DMA->Highest where Highest is an unbalanced zone. Kswapd will reclaim from DMA32 even if it is balanced if Normal is below watermarks. Let N = high_wmark(Normal) + high_wmark(DMA32) Of the last N allocations, some percentage will be allocated from Normal and some from DMA32. The percentage depends on the ratio of the zone sizes and when their watermarks were hit. If Normal is unbalanced, DMA32 will be shrunk by kswapd. If DMA32 is unbalanced only DMA32 will be shrunk. This leads to a difference of ages between DMA32 and Normal. Relatively young pages are then continually rotated and reclaimed from DMA32 due to the higher zone being unbalanced. A debugging patch showed that some PageReadahead pages are reaching the end of the LRU. The number is not very high but it's there. Monitoring of nr_free_pages on a per-zone basis show that there is constant reclaim of the lower zones even when the watermarks should be ok. The iostats showed that without the page there are more pages being read. I believe the difference in sequential read performance is because relatively young pages recently readahead are being reclaimed from the lower zones. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>