On Wed, Jul 26, 2023 at 3:52 AM Ryan Roberts <ryan.roberts@xxxxxxx> wrote:
>
> Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to be
> allocated in large folios of a determined order. All pages of the large
> folio are pte-mapped during the same page fault, significantly reducing
> the number of page faults. The number of per-page operations (e.g. ref
> counting, rmap management lru list management) are also significantly
> reduced since those ops now become per-folio.
>
> The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig,
> which defaults to disabled for now; The long term aim is for this to
> defaut to enabled, but there are some risks around internal
> fragmentation that need to be better understood first.
>
> When enabled, the folio order is determined as such: For a vma, process
> or system that has explicitly disabled THP, we continue to allocate
> order-0. THP is most likely disabled to avoid any possible internal
> fragmentation so we honour that request.
>
> Otherwise, the return value of arch_wants_pte_order() is used. For vmas
> that have not explicitly opted-in to use transparent hugepages (e.g.
> where thp=madvise and the vma does not have MADV_HUGEPAGE), then
> arch_wants_pte_order() is limited to 64K (or PAGE_SIZE, whichever is
> bigger). This allows for a performance boost without requiring any
> explicit opt-in from the workload while limitting internal
> fragmentation.
>
> If the preferred order can't be used (e.g. because the folio would
> breach the bounds of the vma, or because ptes in the region are already
> mapped) then we fall back to a suitable lower order; first
> PAGE_ALLOC_COSTLY_ORDER, then order-0.
>
> arch_wants_pte_order() can be overridden by the architecture if desired.
> Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous
> set of ptes map physically contigious, naturally aligned memory, so this
> mechanism allows the architecture to optimize as required.
>
> Here we add the default implementation of arch_wants_pte_order(), used
> when the architecture does not define it, which returns -1, implying
> that the HW has no preference. In this case, mm will choose it's own
> default order.
>
> Signed-off-by: Ryan Roberts <ryan.roberts@xxxxxxx>
> ---
> include/linux/pgtable.h | 13 ++++
> mm/Kconfig | 10 +++
> mm/memory.c | 166 ++++++++++++++++++++++++++++++++++++----
> 3 files changed, 172 insertions(+), 17 deletions(-)
>
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index 5063b482e34f..2a1d83775837 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
> @@ -313,6 +313,19 @@ static inline bool arch_has_hw_pte_young(void)
> }
> #endif
>
> +#ifndef arch_wants_pte_order
> +/*
> + * Returns preferred folio order for pte-mapped memory. Must be in range [0,
> + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios
> + * to be at least order-2. Negative value implies that the HW has no preference
> + * and mm will choose it's own default order.
> + */
> +static inline int arch_wants_pte_order(void)
> +{
> + return -1;
> +}
> +#endif
> +
> #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR
> static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
> unsigned long address,
> diff --git a/mm/Kconfig b/mm/Kconfig
> index 09130434e30d..fa61ea160447 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -1238,4 +1238,14 @@ config LOCK_MM_AND_FIND_VMA
>
> source "mm/damon/Kconfig"
>
> +config LARGE_ANON_FOLIO
> + bool "Allocate large folios for anonymous memory"
> + depends on TRANSPARENT_HUGEPAGE
> + default n
> + help
> + Use large (bigger than order-0) folios to back anonymous memory where
> + possible, even for pte-mapped memory. This reduces the number of page
> + faults, as well as other per-page overheads to improve performance for
> + many workloads.
> +
> endmenu
> diff --git a/mm/memory.c b/mm/memory.c
> index 01f39e8144ef..64c3f242c49a 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -4050,6 +4050,127 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
> return ret;
> }
>
> +static bool vmf_pte_range_changed(struct vm_fault *vmf, int nr_pages)
> +{
> + int i;
> +
> + if (nr_pages == 1)
> + return vmf_pte_changed(vmf);
> +
> + for (i = 0; i < nr_pages; i++) {
> + if (!pte_none(ptep_get_lockless(vmf->pte + i)))
> + return true;
> + }
> +
> + return false;
> +}
> +
> +#ifdef CONFIG_LARGE_ANON_FOLIO
> +#define ANON_FOLIO_MAX_ORDER_UNHINTED \
> + (ilog2(max_t(unsigned long, SZ_64K, PAGE_SIZE)) - PAGE_SHIFT)
> +
> +static int anon_folio_order(struct vm_area_struct *vma)
> +{
> + int order;
> +
> + /*
> + * If THP is explicitly disabled for either the vma, the process or the
> + * system, then this is very likely intended to limit internal
> + * fragmentation; in this case, don't attempt to allocate a large
> + * anonymous folio.
> + *
> + * Else, if the vma is eligible for thp, allocate a large folio of the
> + * size preferred by the arch. Or if the arch requested a very small
> + * size or didn't request a size, then use PAGE_ALLOC_COSTLY_ORDER,
> + * which still meets the arch's requirements but means we still take
> + * advantage of SW optimizations (e.g. fewer page faults).
> + *
> + * Finally if thp is enabled but the vma isn't eligible, take the
> + * arch-preferred size and limit it to ANON_FOLIO_MAX_ORDER_UNHINTED.
> + * This ensures workloads that have not explicitly opted-in take benefit
> + * while capping the potential for internal fragmentation.
> + */
> +
> + if ((vma->vm_flags & VM_NOHUGEPAGE) ||
> + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags) ||
> + !hugepage_flags_enabled())
> + order = 0;
> + else {
> + order = max(arch_wants_pte_order(), PAGE_ALLOC_COSTLY_ORDER);
> +
> + if (!hugepage_vma_check(vma, vma->vm_flags, false, true, true))
> + order = min(order, ANON_FOLIO_MAX_ORDER_UNHINTED);
> + }
> +
> + return order;
> +}
> +
> +static int alloc_anon_folio(struct vm_fault *vmf, struct folio **folio)
> +{
> + int i;
> + gfp_t gfp;
> + pte_t *pte;
> + unsigned long addr;
> + struct vm_area_struct *vma = vmf->vma;
> + int prefer = anon_folio_order(vma);
> + int orders[] = {
> + prefer,
> + prefer > PAGE_ALLOC_COSTLY_ORDER ? PAGE_ALLOC_COSTLY_ORDER : 0,
> + 0,
> + };
> +
> + *folio = NULL;
> +
> + if (vmf_orig_pte_uffd_wp(vmf))
> + goto fallback;
I think we need to s/vmf_orig_pte_uffd_wp/userfaultfd_armed/ here;
otherwise UFFD would miss VM_UFFD_MISSING/MINOR.
