This is a breakout series from the huge page allocator patches[1].
While testing and benchmarking the series incrementally, as per
reviewer request, it became apparent that there are several sources of
freelist migratetype violations that later patches in the series hid.
Those violations occur when pages of one migratetype end up on the
freelists of another type. This encourages incompatible page mixing
down the line, where allocation requests ask for one migrate type, but
receives pages of another. This defeats the mobility grouping.
The series addresses those causes. The last patch adds type checks on
all freelist movements to rule out any violations. I used these checks
to identify the violations fixed up in the preceding patches.
The series is a breakout, but has merit on its own: Less type mixing
means improved grouping, means less work for compaction, means higher
THP success rate and lower allocation latencies. The results can be
seen in a mixed workload that stresses the machine with a kernel build
job while periodically attempting to allocate batches of THP. The data
is averaged over 50 consecutive defconfig builds:
VANILLA PATCHED-CLEANLISTS
Hugealloc Time median 14642.00 ( +0.00%) 10506.00 ( -28.25%)
Hugealloc Time min 4820.00 ( +0.00%) 4783.00 ( -0.77%)
Hugealloc Time max 6786868.00 ( +0.00%) 6556624.00 ( -3.39%)
Kbuild Real time 240.03 ( +0.00%) 241.45 ( +0.59%)
Kbuild User time 1195.49 ( +0.00%) 1195.69 ( +0.02%)
Kbuild System time 96.44 ( +0.00%) 97.03 ( +0.61%)
THP fault alloc 11490.00 ( +0.00%) 11802.30 ( +2.72%)
THP fault fallback 782.62 ( +0.00%) 478.88 ( -38.76%)
THP fault fail rate % 6.38 ( +0.00%) 3.90 ( -33.52%)
Direct compact stall 297.70 ( +0.00%) 224.56 ( -24.49%)
Direct compact fail 265.98 ( +0.00%) 191.56 ( -27.87%)
Direct compact success 31.72 ( +0.00%) 33.00 ( +3.91%)
Direct compact success rate % 13.11 ( +0.00%) 17.26 ( +29.43%)
Compact daemon scanned migrate 1673661.58 ( +0.00%) 1591682.18 ( -4.90%)
Compact daemon scanned free 2711252.80 ( +0.00%) 2615217.78 ( -3.54%)
Compact direct scanned migrate 384998.62 ( +0.00%) 261689.42 ( -32.03%)
Compact direct scanned free 966308.94 ( +0.00%) 667459.76 ( -30.93%)
Compact migrate scanned daemon % 80.86 ( +0.00%) 83.34 ( +3.02%)
Compact free scanned daemon % 74.41 ( +0.00%) 78.26 ( +5.10%)
Alloc stall 338.06 ( +0.00%) 440.72 ( +30.28%)
Pages kswapd scanned 1356339.42 ( +0.00%) 1402313.42 ( +3.39%)
Pages kswapd reclaimed 581309.08 ( +0.00%) 587956.82 ( +1.14%)
Pages direct scanned 56384.18 ( +0.00%) 141095.04 ( +150.24%)
Pages direct reclaimed 17055.54 ( +0.00%) 22427.96 ( +31.50%)
Pages scanned kswapd % 96.38 ( +0.00%) 93.60 ( -2.86%)
Swap out 41528.00 ( +0.00%) 47969.92 ( +15.51%)
Swap in 6541.42 ( +0.00%) 9093.30 ( +39.01%)
File refaults 127666.50 ( +0.00%) 135766.84 ( +6.34%)
The series is based on v6.5-rc7.
include/linux/mm.h | 18 +-
include/linux/page-isolation.h | 2 +-
include/linux/vmstat.h | 8 -
mm/debug_page_alloc.c | 12 +-
mm/internal.h | 5 -
mm/page_alloc.c | 382 +++++++++++++++++++++------------------
mm/page_isolation.c | 23 ++-
7 files changed, 230 insertions(+), 220 deletions(-)
[1] https://lore.kernel.org/lkml/20230418191313.268131-1-hannes@xxxxxxxxxxx/
