+CC linux-api, please do on further revisions.
Keeping rest of the e-mail.
On 2/17/21 4:48 PM, David Hildenbrand wrote:
> When we manage sparse memory mappings dynamically in user space - also
> sometimes involving MADV_NORESERVE - we want to dynamically populate/
> discard memory inside such a sparse memory region. Example users are
> hypervisors (especially implementing memory ballooning or similar
> technologies like virtio-mem) and memory allocators. In addition, we want
> to fail in a nice way if populating does not succeed because we are out of
> backend memory (which can happen easily with file-based mappings,
> especially tmpfs and hugetlbfs).
>
> While MADV_DONTNEED and FALLOC_FL_PUNCH_HOLE provide us ways to reliably
> discard memory, there is no generic approach to populate ("preallocate")
> memory.
>
> Although mmap() supports MAP_POPULATE, it is not applicable to the concept
> of sparse memory mappings, where we want to do populate/discard
> dynamically and avoid expensive/problematic remappings. In addition,
> we never actually report error during the final populate phase - it is
> best-effort only.
>
> fallocate() can be used to preallocate file-based memory and fail in a safe
> way. However, it is less useful for private mappings on anonymous files
> due to COW semantics. For example, using fallocate() to preallocate memory
> on an anonymous memfd files that are mapped MAP_PRIVATE results in a double
> memory consumption when actually writing via the mapping. In addition,
> fallocate() does not actually populate page tables, so we still always
> have to resolve minor faults on first access.
>
> Because we don't have a proper interface, what applications
> (like QEMU and databases) end up doing is touching (i.e., writing) all
> individual pages. However, it requires expensive signal handling (SIGBUS);
> for example, this is problematic in hypervisors like QEMU where SIGBUS
> handlers might already be used by other subsystems concurrently to e.g,
> handle hardware errors. "Simply" doing preallocation from another thread
> is not that easy.
>
> Let's introduce MADV_POPULATE with the following semantics
> 1. MADV_POPULATED does not work on PROT_NONE and special VMAs. It works
> on everything else.
> 2. Errors during MADV_POPULATED (especially OOM) are reported. If we hit
> hardware errors on pages, ignore them - nothing we really can or
> should do.
> 3. On errors during MADV_POPULATED, some memory might have been
> populated. Callers have to clean up if they care.
> 4. Concurrent changes to the virtual memory layour are tolerated - we
> process each and every PFN only once, though.
> 5. If MADV_POPULATE succeeds, all memory in the range can be accessed
> without SIGBUS. (of course, not if user space changed mappings in the
> meantime or KSM kicked in on anonymous memory).
>
> Although sparse memory mappings are the primary use case, this will
> also be useful for ordinary preallocations where MAP_POPULATE is not
> desired (e.g., in QEMU, where users can trigger preallocation of
> guest RAM after the mapping was created).
>
> Looking at the history, MADV_POPULATE was already proposed in 2013 [1],
> however, the main motivation back than was performance improvements
> (which should also still be the case, but it's a seconary concern).
>
> Basic functionality was tested with:
> - anonymous memory
> - MAP_PRIVATE on anonymous file via memfd
> - MAP_SHARED on anonymous file via memf
> - MAP_PRIVATE on anonymous hugetlbfs file via memfd
> - MAP_SHARED on anonymous hugetlbfs file via memfd
> - MAP_PRIVATE on tmpfs/shmem file (we end up with double memory consumption
> though, as the actual file gets populated with zeroes)
> - MAP_SHARED on tmpfs/shmem file
>
> Note: For populating/preallocating zeroed-out memory while userfaultfd is
> active, it's even faster to use first fallocate() or placing zeroed pages
> via userfaultfd APIs. Otherwise, we'll have to route every fault while
> populating via the userfaultfd handler.
>
> [1] https://lkml.org/lkml/2013/6/27/698
>
> Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
> Cc: Arnd Bergmann <arnd@xxxxxxxx>
> Cc: Michal Hocko <mhocko@xxxxxxxx>
> Cc: Oscar Salvador <osalvador@xxxxxxx>
> Cc: Matthew Wilcox (Oracle) <willy@xxxxxxxxxxxxx>
> Cc: Andrea Arcangeli <aarcange@xxxxxxxxxx>
> Cc: Minchan Kim <minchan@xxxxxxxxxx>
> Cc: Jann Horn <jannh@xxxxxxxxxx>
> Cc: Jason Gunthorpe <jgg@xxxxxxxx>
> Cc: Dave Hansen <dave.hansen@xxxxxxxxx>
> Cc: Hugh Dickins <hughd@xxxxxxxxxx>
> Cc: Rik van Riel <riel@xxxxxxxxxxx>
> Cc: Michael S. Tsirkin <mst@xxxxxxxxxx>
> Cc: Kirill A. Shutemov <kirill.shutemov@xxxxxxxxxxxxxxx>
> Cc: Vlastimil Babka <vbabka@xxxxxxx>
> Cc: Richard Henderson <rth@xxxxxxxxxxx>
> Cc: Ivan Kokshaysky <ink@xxxxxxxxxxxxxxxxxxxx>
> Cc: Matt Turner <mattst88@xxxxxxxxx>
> Cc: Thomas Bogendoerfer <tsbogend@xxxxxxxxxxxxxxxx>
> Cc: "James E.J. Bottomley" <James.Bottomley@xxxxxxxxxxxxxxxxxxxxx>
> Cc: Helge Deller <deller@xxxxxx>
> Cc: Chris Zankel <chris@xxxxxxxxxx>
> Cc: Max Filippov <jcmvbkbc@xxxxxxxxx>
> Cc: linux-alpha@xxxxxxxxxxxxxxx
> Cc: linux-mips@xxxxxxxxxxxxxxx
> Cc: linux-parisc@xxxxxxxxxxxxxxx
> Cc: linux-xtensa@xxxxxxxxxxxxxxxx
> Cc: linux-arch@xxxxxxxxxxxxxxx
> Signed-off-by: David Hildenbrand <david@xxxxxxxxxx>
> ---
>
> If we agree that this makes sense I'll do more testing to see if we
> are missing any return value handling and prepare a man page update to
> document the semantics.
>
> Thoughts?
>
> ---
> arch/alpha/include/uapi/asm/mman.h | 2 +
> arch/mips/include/uapi/asm/mman.h | 2 +
> arch/parisc/include/uapi/asm/mman.h | 2 +
> arch/xtensa/include/uapi/asm/mman.h | 2 +
> include/uapi/asm-generic/mman-common.h | 2 +
> mm/madvise.c | 70 ++++++++++++++++++++++++++
> 6 files changed, 80 insertions(+)
>
> diff --git a/arch/alpha/include/uapi/asm/mman.h b/arch/alpha/include/uapi/asm/mman.h
> index a18ec7f63888..e90eeb5e6cf1 100644
> --- a/arch/alpha/include/uapi/asm/mman.h
> +++ b/arch/alpha/include/uapi/asm/mman.h
> @@ -71,6 +71,8 @@
> #define MADV_COLD 20 /* deactivate these pages */
> #define MADV_PAGEOUT 21 /* reclaim these pages */
>
> +#define MADV_POPULATE 22 /* populate pages */
> +
> /* compatibility flags */
> #define MAP_FILE 0
>
> diff --git a/arch/mips/include/uapi/asm/mman.h b/arch/mips/include/uapi/asm/mman.h
> index 57dc2ac4f8bd..b928becc5308 100644
> --- a/arch/mips/include/uapi/asm/mman.h
> +++ b/arch/mips/include/uapi/asm/mman.h
> @@ -98,6 +98,8 @@
> #define MADV_COLD 20 /* deactivate these pages */
> #define MADV_PAGEOUT 21 /* reclaim these pages */
>
> +#define MADV_POPULATE 22 /* populate pages */
> +
> /* compatibility flags */
> #define MAP_FILE 0
>
> diff --git a/arch/parisc/include/uapi/asm/mman.h b/arch/parisc/include/uapi/asm/mman.h
> index ab78cba446ed..9d3a56044287 100644
> --- a/arch/parisc/include/uapi/asm/mman.h
> +++ b/arch/parisc/include/uapi/asm/mman.h
> @@ -52,6 +52,8 @@
> #define MADV_COLD 20 /* deactivate these pages */
> #define MADV_PAGEOUT 21 /* reclaim these pages */
>
> +#define MADV_POPULATE 22 /* populate pages */
> +
> #define MADV_MERGEABLE 65 /* KSM may merge identical pages */
> #define MADV_UNMERGEABLE 66 /* KSM may not merge identical pages */
>
> diff --git a/arch/xtensa/include/uapi/asm/mman.h b/arch/xtensa/include/uapi/asm/mman.h
> index e5e643752947..3169b1be8920 100644
> --- a/arch/xtensa/include/uapi/asm/mman.h
> +++ b/arch/xtensa/include/uapi/asm/mman.h
> @@ -106,6 +106,8 @@
> #define MADV_COLD 20 /* deactivate these pages */
> #define MADV_PAGEOUT 21 /* reclaim these pages */
>
> +#define MADV_POPULATE 22 /* populate pages */
> +
> /* compatibility flags */
> #define MAP_FILE 0
>
> diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
> index f94f65d429be..fa617fd0d733 100644
> --- a/include/uapi/asm-generic/mman-common.h
> +++ b/include/uapi/asm-generic/mman-common.h
> @@ -72,6 +72,8 @@
> #define MADV_COLD 20 /* deactivate these pages */
> #define MADV_PAGEOUT 21 /* reclaim these pages */
>
> +#define MADV_POPULATE 22 /* populate pages */
> +
> /* compatibility flags */
> #define MAP_FILE 0
>
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 6a660858784b..f76fdd6fcf10 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -53,6 +53,7 @@ static int madvise_need_mmap_write(int behavior)
> case MADV_COLD:
> case MADV_PAGEOUT:
> case MADV_FREE:
> + case MADV_POPULATE:
> return 0;
> default:
> /* be safe, default to 1. list exceptions explicitly */
> @@ -821,6 +822,72 @@ static long madvise_dontneed_free(struct vm_area_struct *vma,
> return -EINVAL;
> }
>
> +static long madvise_populate(struct vm_area_struct *vma,
> + struct vm_area_struct **prev,
> + unsigned long start, unsigned long end)
> +{
> + struct mm_struct *mm = vma->vm_mm;
> + unsigned long tmp_end;
> + int locked = 1;
> + long pages;
> +
> + *prev = vma;
> +
> + while (start < end) {
> + /*
> + * We might have temporarily dropped the lock. For example,
> + * our VMA might have been split.
> + */
> + if (!vma || start >= vma->vm_end) {
> + vma = find_vma(mm, start);
> + if (!vma)
> + return -ENOMEM;
> + }
> +
> + /* Bail out on incompatible VMA types. */
> + if (vma->vm_flags & (VM_IO | VM_PFNMAP) ||
> + !vma_is_accessible(vma)) {
> + return -EINVAL;
> + }
> +
> + /*
> + * Populate pages and take care of VM_LOCKED: simulate user
> + * space access.
> + *
> + * For private, writable mappings, trigger a write fault to
> + * break COW (i.e., shared zeropage). For other mappings (i.e.,
> + * read-only, shared), trigger a read fault.
> + */
> + tmp_end = min_t(unsigned long, end, vma->vm_end);
> + pages = populate_vma_page_range(vma, start, tmp_end, &locked);
> + if (!locked) {
> + mmap_read_lock(mm);
> + *prev = NULL;
> + vma = NULL;
> + }
> + if (pages < 0) {
> + switch (pages) {
> + case -EINTR:
> + case -ENOMEM:
> + return pages;
> + case -EHWPOISON:
> + /* Skip over any poisoned pages. */
> + start += PAGE_SIZE;
> + continue;
> + case -EBUSY:
> + case -EAGAIN:
> + continue;
> + default:
> + pr_warn_once("%s: unhandled return value: %ld\n",
> + __func__, pages);
> + return -ENOMEM;
> + }
> + }
> + start += pages * PAGE_SIZE;
> + }
> + return 0;
> +}
> +
> /*
> * Application wants to free up the pages and associated backing store.
> * This is effectively punching a hole into the middle of a file.
> @@ -934,6 +1001,8 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev,
> case MADV_FREE:
> case MADV_DONTNEED:
> return madvise_dontneed_free(vma, prev, start, end, behavior);
> + case MADV_POPULATE:
> + return madvise_populate(vma, prev, start, end);
> default:
> return madvise_behavior(vma, prev, start, end, behavior);
> }
> @@ -954,6 +1023,7 @@ madvise_behavior_valid(int behavior)
> case MADV_FREE:
> case MADV_COLD:
> case MADV_PAGEOUT:
> + case MADV_POPULATE:
> #ifdef CONFIG_KSM
> case MADV_MERGEABLE:
> case MADV_UNMERGEABLE:
>
