From: Barry Song <v-songbaohua@xxxxxxxx>
Commit 735ecdfaf4e8 ("mm/vmscan: avoid split lazyfree THP during
shrink_folio_list()") prevents the splitting of MADV_FREE'd THP in
madvise.c.
However, those folios are still added to the deferred_split list in
try_to_unmap_one() because we are unmapping PTEs and removing rmap
entries one by one.
Firstly, this has rendered the following counter somewhat confusing,
/sys/kernel/mm/transparent_hugepage/hugepages-size/stats/split_deferred
The split_deferred counter was originally designed to track operations
such as partial unmap or madvise of large folios. However, in practice,
most split_deferred cases arise from memory reclamation of aligned
lazyfree mTHPs as observed by Tangquan. This discrepancy has made
the split_deferred counter highly misleading.
Secondly, this approach is slow because it requires iterating through
each PTE and removing the rmap one by one for a large folio. In fact,
all PTEs of a pte-mapped large folio should be unmapped at once, and
the entire folio should be removed from the rmap as a whole.
Thirdly, it also increases the risk of a race condition where lazyfree
folios are incorrectly set back to swapbacked, as a speculative folio_get
may occur in the shrinker's callback.
deferred_split_scan() might call folio_try_get(folio) since we have
added the folio to split_deferred list while removing rmap for the
1st subpage, and while we are scanning the 2nd to nr_pages PTEs of
this folio in try_to_unmap_one(), the entire mTHP could be
transitioned back to swap-backed because the reference count is
incremented, which can make "ref_count == 1 + map_count" within
try_to_unmap_one() false.
/*
* The only page refs must be one from isolation
* plus the rmap(s) (dropped by discard:).
*/
if (ref_count == 1 + map_count &&
(!folio_test_dirty(folio) ||
...
(vma->vm_flags & VM_DROPPABLE))) {
dec_mm_counter(mm, MM_ANONPAGES);
goto discard;
}
This patchset resolves the issue by marking only genuinely dirty folios
as swap-backed, as suggested by David, and transitioning to batched
unmapping of entire folios in try_to_unmap_one(). Consequently, the
deferred_split count drops to zero, and memory reclamation performance
improves significantly — reclaiming 64KiB lazyfree large folios is now
2.5x faster(The specific data is embedded in the changelog of patch
3/4).
By the way, while the patchset is primarily aimed at PTE-mapped large
folios, Baolin and Lance also found that try_to_unmap_one() handles
lazyfree redirtied PMD-mapped large folios inefficiently — it splits
the PMD into PTEs and iterates over them. This patchset removes the
unnecessary splitting, enabling us to skip redirtied PMD-mapped large
folios 3.5X faster during memory reclamation. (The specific data can
be found in the changelog of patch 4/4).
-v2:
* describle backgrounds, problems more clearly in cover-letter per
Lorenzo Stoakes;
* also handle redirtied pmd-mapped large folios per Baolin and Lance;
* handle some corner cases such as HWPOSION, pte_unused;
* riscv and x86 build issues.
-v1:
https://lore.kernel.org/linux-mm/20250106031711.82855-1-21cnbao@xxxxxxxxx/
Barry Song (4):
mm: Set folio swapbacked iff folios are dirty in try_to_unmap_one
mm: Support tlbbatch flush for a range of PTEs
mm: Support batched unmap for lazyfree large folios during reclamation
mm: Avoid splitting pmd for lazyfree pmd-mapped THP in try_to_unmap
arch/arm64/include/asm/tlbflush.h | 26 +++----
arch/arm64/mm/contpte.c | 2 +-
arch/riscv/include/asm/tlbflush.h | 3 +-
arch/riscv/mm/tlbflush.c | 3 +-
arch/x86/include/asm/tlbflush.h | 3 +-
mm/huge_memory.c | 17 ++++-
mm/rmap.c | 112 ++++++++++++++++++++----------
7 files changed, 111 insertions(+), 55 deletions(-)
--
2.39.3 (Apple Git-146)
