Introduction
--------------------------------
This series provides a mechanism for userspace to induce a collapse of
eligible ranges of memory into transparent hugepages in process context,
thus permitting users to more tightly control their own hugepage
utilization policy at their own expense.
This idea was introduced by David Rientjes[1], and the semantics and
implementation were introduced and discussed in a previous PATCH RFC[2].
Interface
--------------------------------
The proposed interface adds a new madvise(2) mode, MADV_COLLAPSE, and
leverages the new process_madvise(2) call.
process_madvise(2)
Performs a synchronous collapse of the native pages
mapped by the list of iovecs into transparent hugepages.
The system-wide THP sysfs settings as well as the VMA flags of the
memory range being collapsed are both respected for the purposes
of determining THP eligibility.
THP allocation may enter direct reclaim and/or compaction.
When a range spans multiple VMAs, the semantics of the collapse
over of each VMA is independent from the others.
Caller must have CAP_SYS_ADMIN if not acting on self.
Return value follows existing process_madvise(2) conventions. A
“success” indicates that all hugepage-sized/aligned regions
covered by the provided range were either successfully
collapsed, or were already pmd-mapped THPs.
madvise(2)
Equivalent to process_madvise(2) on self, with 0 returned on
“success”.
Current Use-Cases
--------------------------------
(1) Immediately back executable text by THPs. Current support provided
by CONFIG_READ_ONLY_THP_FOR_FS may take a long time on a large
system which might impair services from serving at their full rated
load after (re)starting. Tricks like mremap(2)'ing text onto
anonymous memory to immediately realize iTLB performance prevents
page sharing and demand paging, both of which increase steady state
memory footprint. With MADV_COLLAPSE, we get the best of both
worlds: Peak upfront performance and lower RAM footprints. Note
that subsequent support for file-backed memory is required here.
(2) malloc() implementations that manage memory in hugepage-sized
chunks, but sometimes subrelease memory back to the system in
native-sized chunks via MADV_DONTNEED; zapping the pmd. Later, when
the memory is hot, the implementation could madvise(MADV_COLLAPSE)
to re-back the memory by THPs to regain hugepage coverage and dTLB
performance. TCMalloc is such an implementation that could benefit
from this[3]. A prior study of Google internal workloads during
evaluation of Temeraire, a hugepage-aware enhancement to TCMalloc,
showed that nearly 20% of all cpu cycles were spent in dTLB stalls,
and that increasing hugepage coverage by even small amount can help
with that[4].
Future work
--------------------------------
Only private anonymous memory is supported by this series. File and
shmem memory support will be added later.
One possible user of this functionality is a userspace agent that
attempts to optimize THP utilization system-wide by allocating THPs
based on, for example, task priority, task performance requirements, or
heatmaps. For the latter, one idea that has already surfaced is using
DAMON to identify hot regions, and driving THP collapse through a new
DAMOS_COLLAPSE scheme[5].
Sequence of Patches
--------------------------------
Patches 1-7 perform refactoring of collapse logic within khugepaged.c,
introduce the notion of a collapse context, and add needed hooks to
specialize per-collapse context functionality.
Patches 8-10 introduces MADV_COLLAPSE, does some renaming, and adds
process_madivse(2) support.
Patches 11-13 add selftests to test the new functionality.
Applies against next-20220502.
Changelog
--------------------------------
v3 -> v4:
* 'mm/khugepaged: record SCAN_PMD_MAPPED when scan_pmd() finds THP'
-> Dropped pmd_none() check from find_pmd_or_thp_or_none()
-> Moved SCAN_PMD_MAPPED after SCAN_PMD_NULL
-> Dropped <lkp@xxxxxxxxx> from sign-offs
* 'mm/khugepaged: add struct collapse_control'
-> Updated commit description and some code comments
-> Removed extra brackets added in khugepaged_find_target_node()
* Added 'mm/khugepaged: dedup hugepage allocation and charging code'
* 'mm/khugepaged: make hugepage allocation context-specific'
-> Has been majorly reworked to replace ->gfp() and ->alloc_hpage()
struct collapse_control hooks with a ->alloc_charge_hpage() hook
which makes node-allocation, gfp flags, node scheduling, hpage
allocation, and accounting/charging context-specific.
-> Dropped <lkp@xxxxxxxxx> from sign-offs
* Added 'mm/khugepaged: pipe enum scan_result codes back to callers'
-> Replaces 'mm/khugepaged: add struct collapse_result'
* Dropped 'mm/khugepaged: add struct collapse_result'
* 'mm/khugepaged: add flag to ignore khugepaged_max_ptes_*'
-> Moved before 'mm/madvise: introduce MADV_COLLAPSE sync hugepage
collapse'
* 'mm/khugepaged: add flag to ignore page young/referenced requirement'
-> Moved before 'mm/madvise: introduce MADV_COLLAPSE sync hugepage
collapse'
* 'mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse'
-> Moved struct collapse_control* argument to end of alloc_hpage()
-> Some refactoring to rebase on top changes to struct
collapse_control hook changes and other previous commits.
-> Reworded commit description
-> Dropped <lkp@xxxxxxxxx> from sign-offs
* 'mm/khugepaged: rename prefix of shared collapse functions'
-> Renamed from 'mm/khugepaged: remove khugepaged prefix from shared
collapse functions'
-> Instead of dropping "khugepaged_" prefix, replace with
"hpage_collapse_"
-> Dropped <lkp@xxxxxxxxx> from sign-offs
* Rebased onto next-20220502
v2 -> v3:
* Collapse semantics have changed: the gfp flags used for hugepage allocation
now are independent of khugepaged.
* Cover-letter: add primary use-cases and update description of collapse
semantics.
* 'mm/khugepaged: make hugepage allocation context-specific'
-> Added .gfp operation to struct collapse_control
* 'mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse'
-> Added madvise context .gfp implementation.
-> Set scan_result appropriately on early exit due to mm exit or vma
vma revalidation.
-> Reword patch description
* Rebased onto next-20220426
v1 -> v2:
* Cover-letter clarification and added RFC -> v1 notes
* Fixes issues reported by kernel test robot <lkp@xxxxxxxxx>
* 'mm/khugepaged: record SCAN_PMD_MAPPED when scan_pmd() finds THP'
-> Fixed mixed code/declarations
* 'mm/khugepaged: make hugepage allocation context-specific'
-> Fixed bad function signature in !NUMA && TRANSPARENT_HUGEPAGE configs
-> Added doc comment to retract_page_tables() for "cc"
* 'mm/khugepaged: add struct collapse_result'
-> Added doc comment to retract_page_tables() for "cr"
* 'mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse'
-> Added MADV_COLLAPSE definitions for alpha, mips, parisc, xtensa
-> Moved an "#ifdef NUMA" so that khugepaged_find_target_node() is
defined in !NUMA && TRANSPARENT_HUGEPAGE configs.
* 'mm/khugepaged: remove khugepaged prefix from shared collapse'
functions
-> Removed khugepaged prefix from khugepaged_find_target_node on L914
* Rebased onto next-20220414
RFC -> v1:
* The series was significantly reworked from RFC and most patches are entirely
new or reworked.
* Collapse eligibility criteria has changed: MADV_COLLAPSE now respects
VM_NOHUGEPAGE.
* Collapse semantics have changed: the gfp flags used for hugepage allocation
now match that of khugepaged for the same VMA, instead of the gfp flags used
at-fault for calling process for the VMA.
* Collapse semantics have changed: The collapse semantics for multiple VMAs
spanning a single MADV_COLLAPSE call are now independent, whereas before the
idea was to allow direct reclaim/compaction if any spanned VMA permitted so.
* The process_madvise(2) flags, MADV_F_COLLAPSE_LIMITS and
MADV_F_COLLAPSE_DEFRAG have been removed.
* Implementation change: the RFC implemented collapse over a range of hugepages
in a batched-fashion with the aim of doing multiple page table updates inside
a single mmap_lock write. This has been changed, and the implementation now
collapses each hugepage-aligned/sized region iteratively. This was motivated
by an experiment which showed that, when multiple threads were concurrently
faulting during a MADV_COLLAPSE operation, mean and tail latency to acquire
mmap_lock in read for threads in the fault patch was improved by using a batch
size of 1 (batch sizes of 1, 8, 16, 32 were tested)[6].
* Added: If a collapse operation fails because a page isn't found on the LRU, do
a lru_add_drain_all() and retry.
* Added: selftests
[1] https://lore.kernel.org/all/C8C89F13-3F04-456B-BA76-DE2C378D30BF@xxxxxxxxxx/
[2] https://lore.kernel.org/linux-mm/20220308213417.1407042-1-zokeefe@xxxxxxxxxx/
[3] https://github.com/google/tcmalloc/tree/master/tcmalloc
[4] https://research.google/pubs/pub50370/
[5] https://lore.kernel.org/lkml/bcc8d9a0-81d-5f34-5e4-fcc28eb7ce@xxxxxxxxxx/T/
[6] https://lore.kernel.org/linux-mm/CAAa6QmRc76n-dspGT7UK8DkaqZAOz-CkCsME1V7KGtQ6Yt2FqA@xxxxxxxxxxxxxx/
Zach O'Keefe (13):
mm/khugepaged: separate hugepage preallocation and cleanup
mm/khugepaged: record SCAN_PMD_MAPPED when scan_pmd() finds THP
mm/khugepaged: add struct collapse_control
mm/khugepaged: make hugepage allocation context-specific
mm/khugepaged: add struct collapse_result
mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse
mm/khugepaged: remove khugepaged prefix from shared collapse functions
mm/khugepaged: add flag to ignore khugepaged_max_ptes_*
mm/khugepaged: add flag to ignore page young/referenced requirement
mm/madvise: add MADV_COLLAPSE to process_madvise()
selftests/vm: modularize collapse selftests
selftests/vm: add MADV_COLLAPSE collapse context to selftests
selftests/vm: add test to verify recollapse of THPs
include/linux/huge_mm.h | 12 +
include/trace/events/huge_memory.h | 5 +-
include/uapi/asm-generic/mman-common.h | 2 +
mm/internal.h | 1 +
mm/khugepaged.c | 598 ++++++++++++++++--------
mm/madvise.c | 11 +-
mm/rmap.c | 15 +-
tools/testing/selftests/vm/khugepaged.c | 417 +++++++++++------
8 files changed, 702 insertions(+), 359 deletions(-)
--
2.35.1.1178.g4f1659d476-goog
