On Mon, Apr 08, 2024 at 05:30:04PM +0800, Baolin Wang wrote: > > > On 2024/3/21 02:02, Johannes Weiner wrote: > > V4: > > - fixed !pcp_order_allowed() case in free_unref_folios() > > - reworded the patch 0 changelog a bit for the git log > > - rebased to mm-everything-2024-03-19-23-01 > > - runtime-tested again with various CONFIG_DEBUG_FOOs enabled > > > > --- > > > > The page allocator's mobility grouping is intended to keep unmovable > > pages separate from reclaimable/compactable ones to allow on-demand > > defragmentation for higher-order allocations and huge pages. > > > > Currently, there are several places where accidental type mixing > > occurs: an allocation asks for a page of a certain migratetype and > > receives another. This ruins pageblocks for compaction, which in turn > > makes allocating huge pages more expensive and less reliable. > > > > The series addresses those causes. The last patch adds type checks on > > all freelist movements to prevent new violations being introduced. > > > > The benefits can be seen in a mixed workload that stresses the machine > > with a memcache-type workload and a kernel build job while > > periodically attempting to allocate batches of THP. The following data > > is aggregated over 50 consecutive defconfig builds: > > > > VANILLA PATCHED > > Hugealloc Time mean 165843.93 ( +0.00%) 113025.88 ( -31.85%) > > Hugealloc Time stddev 158957.35 ( +0.00%) 114716.07 ( -27.83%) > > Kbuild Real time 310.24 ( +0.00%) 300.73 ( -3.06%) > > Kbuild User time 1271.13 ( +0.00%) 1259.42 ( -0.92%) > > Kbuild System time 582.02 ( +0.00%) 559.79 ( -3.81%) > > THP fault alloc 30585.14 ( +0.00%) 40853.62 ( +33.57%) > > THP fault fallback 36626.46 ( +0.00%) 26357.62 ( -28.04%) > > THP fault fail rate % 54.49 ( +0.00%) 39.22 ( -27.53%) > > Pagealloc fallback 1328.00 ( +0.00%) 1.00 ( -99.85%) > > Pagealloc type mismatch 181009.50 ( +0.00%) 0.00 ( -100.00%) > > Direct compact stall 434.56 ( +0.00%) 257.66 ( -40.61%) > > Direct compact fail 421.70 ( +0.00%) 249.94 ( -40.63%) > > Direct compact success 12.86 ( +0.00%) 7.72 ( -37.09%) > > Direct compact success rate % 2.86 ( +0.00%) 2.82 ( -0.96%) > > Compact daemon scanned migrate 3370059.62 ( +0.00%) 3612054.76 ( +7.18%) > > Compact daemon scanned free 7718439.20 ( +0.00%) 5386385.02 ( -30.21%) > > Compact direct scanned migrate 309248.62 ( +0.00%) 176721.04 ( -42.85%) > > Compact direct scanned free 433582.84 ( +0.00%) 315727.66 ( -27.18%) > > Compact migrate scanned daemon % 91.20 ( +0.00%) 94.48 ( +3.56%) > > Compact free scanned daemon % 94.58 ( +0.00%) 94.42 ( -0.16%) > > Compact total migrate scanned 3679308.24 ( +0.00%) 3788775.80 ( +2.98%) > > Compact total free scanned 8152022.04 ( +0.00%) 5702112.68 ( -30.05%) > > Alloc stall 872.04 ( +0.00%) 5156.12 ( +490.71%) > > Pages kswapd scanned 510645.86 ( +0.00%) 3394.94 ( -99.33%) > > Pages kswapd reclaimed 134811.62 ( +0.00%) 2701.26 ( -98.00%) > > Pages direct scanned 99546.06 ( +0.00%) 376407.52 ( +278.12%) > > Pages direct reclaimed 62123.40 ( +0.00%) 289535.70 ( +366.06%) > > Pages total scanned 610191.92 ( +0.00%) 379802.46 ( -37.76%) > > Pages scanned kswapd % 76.36 ( +0.00%) 0.10 ( -98.58%) > > Swap out 12057.54 ( +0.00%) 15022.98 ( +24.59%) > > Swap in 209.16 ( +0.00%) 256.48 ( +22.52%) > > File refaults 17701.64 ( +0.00%) 11765.40 ( -33.53%) > > > > Huge page success rate is higher, allocation latencies are shorter and > > more predictable. > > > > Stealing (fallback) rate is drastically reduced. Notably, while the > > vanilla kernel keeps doing fallbacks on an ongoing basis, the patched > > kernel enters a steady state once the distribution of block types is > > adequate for the workload. Steals over 50 runs: > > > > VANILLA PATCHED > > 1504.0 227.0 > > 1557.0 6.0 > > 1391.0 13.0 > > 1080.0 26.0 > > 1057.0 40.0 > > 1156.0 6.0 > > 805.0 46.0 > > 736.0 20.0 > > 1747.0 2.0 > > 1699.0 34.0 > > 1269.0 13.0 > > 1858.0 12.0 > > 907.0 4.0 > > 727.0 2.0 > > 563.0 2.0 > > 3094.0 2.0 > > 10211.0 3.0 > > 2621.0 1.0 > > 5508.0 2.0 > > 1060.0 2.0 > > 538.0 3.0 > > 5773.0 2.0 > > 2199.0 0.0 > > 3781.0 2.0 > > 1387.0 1.0 > > 4977.0 0.0 > > 2865.0 1.0 > > 1814.0 1.0 > > 3739.0 1.0 > > 6857.0 0.0 > > 382.0 0.0 > > 407.0 1.0 > > 3784.0 0.0 > > 297.0 0.0 > > 298.0 0.0 > > 6636.0 0.0 > > 4188.0 0.0 > > 242.0 0.0 > > 9960.0 0.0 > > 5816.0 0.0 > > 354.0 0.0 > > 287.0 0.0 > > 261.0 0.0 > > 140.0 1.0 > > 2065.0 0.0 > > 312.0 0.0 > > 331.0 0.0 > > 164.0 0.0 > > 465.0 1.0 > > 219.0 0.0 > > > > Type mismatches are down too. Those count every time an allocation > > request asks for one migratetype and gets another. This can still > > occur minimally in the patched kernel due to non-stealing fallbacks, > > but it's quite rare and follows the pattern of overall fallbacks - > > once the block type distribution settles, mismatches cease as well: > > > > VANILLA: PATCHED: > > 182602.0 268.0 > > 135794.0 20.0 > > 88619.0 19.0 > > 95973.0 0.0 > > 129590.0 0.0 > > 129298.0 0.0 > > 147134.0 0.0 > > 230854.0 0.0 > > 239709.0 0.0 > > 137670.0 0.0 > > 132430.0 0.0 > > 65712.0 0.0 > > 57901.0 0.0 > > 67506.0 0.0 > > 63565.0 4.0 > > 34806.0 0.0 > > 42962.0 0.0 > > 32406.0 0.0 > > 38668.0 0.0 > > 61356.0 0.0 > > 57800.0 0.0 > > 41435.0 0.0 > > 83456.0 0.0 > > 65048.0 0.0 > > 28955.0 0.0 > > 47597.0 0.0 > > 75117.0 0.0 > > 55564.0 0.0 > > 38280.0 0.0 > > 52404.0 0.0 > > 26264.0 0.0 > > 37538.0 0.0 > > 19671.0 0.0 > > 30936.0 0.0 > > 26933.0 0.0 > > 16962.0 0.0 > > 44554.0 0.0 > > 46352.0 0.0 > > 24995.0 0.0 > > 35152.0 0.0 > > 12823.0 0.0 > > 21583.0 0.0 > > 18129.0 0.0 > > 31693.0 0.0 > > 28745.0 0.0 > > 33308.0 0.0 > > 31114.0 0.0 > > 35034.0 0.0 > > 12111.0 0.0 > > 24885.0 0.0 > > > > Compaction work is markedly reduced despite much better THP rates. > > > > In the vanilla kernel, reclaim seems to have been driven primarily by > > watermark boosting that happens as a result of fallbacks. With those > > all but eliminated, watermarks average lower and kswapd does less > > work. The uptick in direct reclaim is because THP requests have to > > fend for themselves more often - which is intended policy right > > now. Aggregate reclaim activity is lowered significantly, though. > > > > --- > > With my 2 fixes, the whole series works well on my platform, so please > feel free to add: > Tested-by: Baolin Wang <baolin.wang@xxxxxxxxxxxxxxxxx> Very much appreciate your testing and the two fixes. Thank you!
