Re: [PATCH 06/10] mm: sys_remap_anon_pages

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



[CC+=linux-api@]

Hi Andrea,

On Wed, Jul 2, 2014 at 6:50 PM, Andrea Arcangeli <aarcange@xxxxxxxxxx> wrote:
> This new syscall will move anon pages across vmas, atomically and
> without touching the vmas.

Please CC linux-api on patches that change the API/ABI. (See
https://www.kernel.org/doc/man-pages/linux-api-ml.html)

Cheers,

Michael


> It only works on non shared anonymous pages because those can be
> relocated without generating non linear anon_vmas in the rmap code.
>
> It is the ideal mechanism to handle userspace page faults. Normally
> the destination vma will have VM_USERFAULT set with
> madvise(MADV_USERFAULT) while the source vma will normally have
> VM_DONTCOPY set with madvise(MADV_DONTFORK).
>
> MADV_DONTFORK set in the source vma avoids remap_anon_pages to fail if
> the process forks during the userland page fault.
>
> The thread triggering the sigbus signal handler by touching an
> unmapped hole in the MADV_USERFAULT region, should take care to
> receive the data belonging in the faulting virtual address in the
> source vma. The data can come from the network, storage or any other
> I/O device. After the data has been safely received in the private
> area in the source vma, it will call remap_anon_pages to map the page
> in the faulting address in the destination vma atomically. And finally
> it will return from the signal handler.
>
> It is an alternative to mremap.
>
> It only works if the vma protection bits are identical from the source
> and destination vma.
>
> It can remap non shared anonymous pages within the same vma too.
>
> If the source virtual memory range has any unmapped holes, or if the
> destination virtual memory range is not a whole unmapped hole,
> remap_anon_pages will fail respectively with -ENOENT or -EEXIST. This
> provides a very strict behavior to avoid any chance of memory
> corruption going unnoticed if there are userland race conditions. Only
> one thread should resolve the userland page fault at any given time
> for any given faulting address. This means that if two threads try to
> both call remap_anon_pages on the same destination address at the same
> time, the second thread will get an explicit error from this syscall.
>
> The syscall retval will return "len" is succesful. The syscall however
> can be interrupted by fatal signals or errors. If interrupted it will
> return the number of bytes successfully remapped before the
> interruption if any, or the negative error if none. It will never
> return zero. Either it will return an error or an amount of bytes
> successfully moved. If the retval reports a "short" remap, the
> remap_anon_pages syscall should be repeated by userland with
> src+retval, dst+reval, len-retval if it wants to know about the error
> that interrupted it.
>
> The RAP_ALLOW_SRC_HOLES flag can be specified to prevent -ENOENT
> errors to materialize if there are holes in the source virtual range
> that is being remapped. The holes will be accounted as successfully
> remapped in the retval of the syscall. This is mostly useful to remap
> hugepage naturally aligned virtual regions without knowing if there
> are transparent hugepage in the regions or not, but preventing the
> risk of having to split the hugepmd during the remap.
>
> The main difference with mremap is that if used to fill holes in
> unmapped anonymous memory vmas (if used in combination with
> MADV_USERFAULT) remap_anon_pages won't create lots of unmergeable
> vmas. mremap instead would create lots of vmas (because of non linear
> vma->vm_pgoff) leading to -ENOMEM failures (the number of vmas is
> limited).
>
> MADV_USERFAULT and remap_anon_pages() can be tested with a program
> like below:
>
> ===
>  #define _GNU_SOURCE
>  #include <sys/mman.h>
>  #include <pthread.h>
>  #include <strings.h>
>  #include <stdlib.h>
>  #include <unistd.h>
>  #include <stdio.h>
>  #include <errno.h>
>  #include <string.h>
>  #include <signal.h>
>  #include <sys/syscall.h>
>  #include <sys/types.h>
>
>  #define USE_USERFAULT
>  #define THP
>
>  #define MADV_USERFAULT 18
>
>  #define SIZE (1024*1024*1024)
>
>  #define SYS_remap_anon_pages 317
>
>  static volatile unsigned char *c, *tmp;
>
>  void userfault_sighandler(int signum, siginfo_t *info, void *ctx)
>  {
>         unsigned char *addr = info->si_addr;
>         int len = 4096;
>         int ret;
>
>         addr = (unsigned char *) ((unsigned long) addr & ~((getpagesize())-1));
>  #ifdef THP
>         addr = (unsigned char *) ((unsigned long) addr & ~((2*1024*1024)-1));
>         len = 2*1024*1024;
>  #endif
>         if (addr >= c && addr < c + SIZE) {
>                 unsigned long offset = addr - c;
>                 ret = syscall(SYS_remap_anon_pages, c+offset, tmp+offset, len, 0);
>                 if (ret != len)
>                         perror("sigbus remap_anon_pages"), exit(1);
>                 //printf("sigbus offset %lu\n", offset);
>                 return;
>         }
>
>         printf("sigbus error addr %p c %p tmp %p\n", addr, c, tmp), exit(1);
>  }
>
>  int main()
>  {
>         struct sigaction sa;
>         int ret;
>         unsigned long i;
>  #ifndef THP
>         /*
>          * Fails with THP due lack of alignment because of memset
>          * pre-filling the destination
>          */
>         c = mmap(0, SIZE, PROT_READ|PROT_WRITE,
>                  MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
>         if (c == MAP_FAILED)
>                 perror("mmap"), exit(1);
>         tmp = mmap(0, SIZE, PROT_READ|PROT_WRITE,
>                    MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
>         if (tmp == MAP_FAILED)
>                 perror("mmap"), exit(1);
>  #else
>         ret = posix_memalign((void **)&c, 2*1024*1024, SIZE);
>         if (ret)
>                 perror("posix_memalign"), exit(1);
>         ret = posix_memalign((void **)&tmp, 2*1024*1024, SIZE);
>         if (ret)
>                 perror("posix_memalign"), exit(1);
>  #endif
>         /*
>          * MADV_USERFAULT must run before memset, to avoid THP 2m
>          * faults to map memory into "tmp", if "tmp" isn't allocated
>          * with hugepage alignment.
>          */
>         if (madvise((void *)c, SIZE, MADV_USERFAULT))
>                 perror("madvise"), exit(1);
>         memset((void *)tmp, 0xaa, SIZE);
>
>         sa.sa_sigaction = userfault_sighandler;
>         sigemptyset(&sa.sa_mask);
>         sa.sa_flags = SA_SIGINFO;
>         sigaction(SIGBUS, &sa, NULL);
>
>  #ifndef USE_USERFAULT
>         ret = syscall(SYS_remap_anon_pages, c, tmp, SIZE, 0);
>         if (ret != SIZE)
>                 perror("remap_anon_pages"), exit(1);
>  #endif
>
>         for (i = 0; i < SIZE; i += 4096) {
>                 if ((i/4096) % 2) {
>                         /* exercise read and write MADV_USERFAULT */
>                         c[i+1] = 0xbb;
>                 }
>                 if (c[i] != 0xaa)
>                         printf("error %x offset %lu\n", c[i], i), exit(1);
>         }
>         printf("remap_anon_pages functions correctly\n");
>
>         return 0;
>  }
> ===
>
> Signed-off-by: Andrea Arcangeli <aarcange@xxxxxxxxxx>
> ---
>  arch/x86/syscalls/syscall_32.tbl |   1 +
>  arch/x86/syscalls/syscall_64.tbl |   1 +
>  include/linux/huge_mm.h          |   7 +
>  include/linux/syscalls.h         |   4 +
>  kernel/sys_ni.c                  |   1 +
>  mm/fremap.c                      | 477 +++++++++++++++++++++++++++++++++++++++
>  mm/huge_memory.c                 | 110 +++++++++
>  7 files changed, 601 insertions(+)
>
> diff --git a/arch/x86/syscalls/syscall_32.tbl b/arch/x86/syscalls/syscall_32.tbl
> index d6b8679..08bc856 100644
> --- a/arch/x86/syscalls/syscall_32.tbl
> +++ b/arch/x86/syscalls/syscall_32.tbl
> @@ -360,3 +360,4 @@
>  351    i386    sched_setattr           sys_sched_setattr
>  352    i386    sched_getattr           sys_sched_getattr
>  353    i386    renameat2               sys_renameat2
> +354    i386    remap_anon_pages        sys_remap_anon_pages
> diff --git a/arch/x86/syscalls/syscall_64.tbl b/arch/x86/syscalls/syscall_64.tbl
> index ec255a1..37bd179 100644
> --- a/arch/x86/syscalls/syscall_64.tbl
> +++ b/arch/x86/syscalls/syscall_64.tbl
> @@ -323,6 +323,7 @@
>  314    common  sched_setattr           sys_sched_setattr
>  315    common  sched_getattr           sys_sched_getattr
>  316    common  renameat2               sys_renameat2
> +317    common  remap_anon_pages        sys_remap_anon_pages
>
>  #
>  # x32-specific system call numbers start at 512 to avoid cache impact
> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> index 3a2c57e..9a37dd5 100644
> --- a/include/linux/huge_mm.h
> +++ b/include/linux/huge_mm.h
> @@ -33,6 +33,13 @@ extern int move_huge_pmd(struct vm_area_struct *vma,
>  extern int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
>                         unsigned long addr, pgprot_t newprot,
>                         int prot_numa);
> +extern int remap_anon_pages_huge_pmd(struct mm_struct *mm,
> +                                    pmd_t *dst_pmd, pmd_t *src_pmd,
> +                                    pmd_t dst_pmdval,
> +                                    struct vm_area_struct *dst_vma,
> +                                    struct vm_area_struct *src_vma,
> +                                    unsigned long dst_addr,
> +                                    unsigned long src_addr);
>
>  enum transparent_hugepage_flag {
>         TRANSPARENT_HUGEPAGE_FLAG,
> diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
> index b0881a0..19edb00 100644
> --- a/include/linux/syscalls.h
> +++ b/include/linux/syscalls.h
> @@ -447,6 +447,10 @@ asmlinkage long sys_mremap(unsigned long addr,
>  asmlinkage long sys_remap_file_pages(unsigned long start, unsigned long size,
>                         unsigned long prot, unsigned long pgoff,
>                         unsigned long flags);
> +asmlinkage long sys_remap_anon_pages(unsigned long dst_start,
> +                                    unsigned long src_start,
> +                                    unsigned long len,
> +                                    unsigned long flags);
>  asmlinkage long sys_msync(unsigned long start, size_t len, int flags);
>  asmlinkage long sys_fadvise64(int fd, loff_t offset, size_t len, int advice);
>  asmlinkage long sys_fadvise64_64(int fd, loff_t offset, loff_t len, int advice);
> diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c
> index 36441b5..6fc1aca 100644
> --- a/kernel/sys_ni.c
> +++ b/kernel/sys_ni.c
> @@ -177,6 +177,7 @@ cond_syscall(sys_mincore);
>  cond_syscall(sys_madvise);
>  cond_syscall(sys_mremap);
>  cond_syscall(sys_remap_file_pages);
> +cond_syscall(sys_remap_anon_pages);
>  cond_syscall(compat_sys_move_pages);
>  cond_syscall(compat_sys_migrate_pages);
>
> diff --git a/mm/fremap.c b/mm/fremap.c
> index 1e509f7..9337637 100644
> --- a/mm/fremap.c
> +++ b/mm/fremap.c
> @@ -310,3 +310,480 @@ void double_pt_unlock(spinlock_t *ptl1,
>         if (ptl1 != ptl2)
>                 spin_unlock(ptl2);
>  }
> +
> +#define RAP_ALLOW_SRC_HOLES (1UL<<0)
> +
> +/*
> + * The mmap_sem for reading is held by the caller. Just move the page
> + * from src_pmd to dst_pmd if possible, and return true if succeeded
> + * in moving the page.
> + */
> +static int remap_anon_pages_pte(struct mm_struct *mm,
> +                               pte_t *dst_pte, pte_t *src_pte, pmd_t *src_pmd,
> +                               struct vm_area_struct *dst_vma,
> +                               struct vm_area_struct *src_vma,
> +                               unsigned long dst_addr,
> +                               unsigned long src_addr,
> +                               spinlock_t *dst_ptl,
> +                               spinlock_t *src_ptl,
> +                               unsigned long flags)
> +{
> +       struct page *src_page;
> +       swp_entry_t entry;
> +       pte_t orig_src_pte, orig_dst_pte;
> +       struct anon_vma *src_anon_vma, *dst_anon_vma;
> +
> +       spin_lock(dst_ptl);
> +       orig_dst_pte = *dst_pte;
> +       spin_unlock(dst_ptl);
> +       if (!pte_none(orig_dst_pte))
> +               return -EEXIST;
> +
> +       spin_lock(src_ptl);
> +       orig_src_pte = *src_pte;
> +       spin_unlock(src_ptl);
> +       if (pte_none(orig_src_pte)) {
> +               if (!(flags & RAP_ALLOW_SRC_HOLES))
> +                       return -ENOENT;
> +               else
> +                       /* nothing to do to remap an hole */
> +                       return 0;
> +       }
> +
> +       if (pte_present(orig_src_pte)) {
> +               /*
> +                * Pin the page while holding the lock to be sure the
> +                * page isn't freed under us
> +                */
> +               spin_lock(src_ptl);
> +               if (!pte_same(orig_src_pte, *src_pte)) {
> +                       spin_unlock(src_ptl);
> +                       return -EAGAIN;
> +               }
> +               src_page = vm_normal_page(src_vma, src_addr, orig_src_pte);
> +               if (!src_page || !PageAnon(src_page) ||
> +                   page_mapcount(src_page) != 1) {
> +                       spin_unlock(src_ptl);
> +                       return -EBUSY;
> +               }
> +
> +               get_page(src_page);
> +               spin_unlock(src_ptl);
> +
> +               /* block all concurrent rmap walks */
> +               lock_page(src_page);
> +
> +               /*
> +                * page_referenced_anon walks the anon_vma chain
> +                * without the page lock. Serialize against it with
> +                * the anon_vma lock, the page lock is not enough.
> +                */
> +               src_anon_vma = page_get_anon_vma(src_page);
> +               if (!src_anon_vma) {
> +                       /* page was unmapped from under us */
> +                       unlock_page(src_page);
> +                       put_page(src_page);
> +                       return -EAGAIN;
> +               }
> +               anon_vma_lock_write(src_anon_vma);
> +
> +               double_pt_lock(dst_ptl, src_ptl);
> +
> +               if (!pte_same(*src_pte, orig_src_pte) ||
> +                   !pte_same(*dst_pte, orig_dst_pte) ||
> +                   page_mapcount(src_page) != 1) {
> +                       double_pt_unlock(dst_ptl, src_ptl);
> +                       anon_vma_unlock_write(src_anon_vma);
> +                       put_anon_vma(src_anon_vma);
> +                       unlock_page(src_page);
> +                       put_page(src_page);
> +                       return -EAGAIN;
> +               }
> +
> +               BUG_ON(!PageAnon(src_page));
> +               /* the PT lock is enough to keep the page pinned now */
> +               put_page(src_page);
> +
> +               dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
> +               ACCESS_ONCE(src_page->mapping) = ((struct address_space *)
> +                                                 dst_anon_vma);
> +               ACCESS_ONCE(src_page->index) = linear_page_index(dst_vma,
> +                                                                dst_addr);
> +
> +               if (!pte_same(ptep_clear_flush(src_vma, src_addr, src_pte),
> +                             orig_src_pte))
> +                       BUG();
> +
> +               orig_dst_pte = mk_pte(src_page, dst_vma->vm_page_prot);
> +               orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte),
> +                                            dst_vma);
> +
> +               set_pte_at(mm, dst_addr, dst_pte, orig_dst_pte);
> +
> +               double_pt_unlock(dst_ptl, src_ptl);
> +
> +               anon_vma_unlock_write(src_anon_vma);
> +               put_anon_vma(src_anon_vma);
> +
> +               /* unblock rmap walks */
> +               unlock_page(src_page);
> +
> +               mmu_notifier_invalidate_page(mm, src_addr);
> +       } else {
> +               if (pte_file(orig_src_pte))
> +                       return -EFAULT;
> +
> +               entry = pte_to_swp_entry(orig_src_pte);
> +               if (non_swap_entry(entry)) {
> +                       if (is_migration_entry(entry)) {
> +                               migration_entry_wait(mm, src_pmd, src_addr);
> +                               return -EAGAIN;
> +                       }
> +                       return -EFAULT;
> +               }
> +
> +               if (swp_entry_swapcount(entry) != 1)
> +                       return -EBUSY;
> +
> +               double_pt_lock(dst_ptl, src_ptl);
> +
> +               if (!pte_same(*src_pte, orig_src_pte) ||
> +                   !pte_same(*dst_pte, orig_dst_pte) ||
> +                   swp_entry_swapcount(entry) != 1) {
> +                       double_pt_unlock(dst_ptl, src_ptl);
> +                       return -EAGAIN;
> +               }
> +
> +               if (pte_val(ptep_get_and_clear(mm, src_addr, src_pte)) !=
> +                   pte_val(orig_src_pte))
> +                       BUG();
> +               set_pte_at(mm, dst_addr, dst_pte, orig_src_pte);
> +
> +               double_pt_unlock(dst_ptl, src_ptl);
> +       }
> +
> +       return 0;
> +}
> +
> +static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address)
> +{
> +       pgd_t *pgd;
> +       pud_t *pud;
> +       pmd_t *pmd = NULL;
> +
> +       pgd = pgd_offset(mm, address);
> +       pud = pud_alloc(mm, pgd, address);
> +       if (pud)
> +               /*
> +                * Note that we didn't run this because the pmd was
> +                * missing, the *pmd may be already established and in
> +                * turn it may also be a trans_huge_pmd.
> +                */
> +               pmd = pmd_alloc(mm, pud, address);
> +       return pmd;
> +}
> +
> +/**
> + * sys_remap_anon_pages - remap arbitrary anonymous pages of an existing vma
> + * @dst_start: start of the destination virtual memory range
> + * @src_start: start of the source virtual memory range
> + * @len: length of the virtual memory range
> + *
> + * sys_remap_anon_pages remaps arbitrary anonymous pages atomically in
> + * zero copy. It only works on non shared anonymous pages because
> + * those can be relocated without generating non linear anon_vmas in
> + * the rmap code.
> + *
> + * It is the ideal mechanism to handle userspace page faults. Normally
> + * the destination vma will have VM_USERFAULT set with
> + * madvise(MADV_USERFAULT) while the source vma will have VM_DONTCOPY
> + * set with madvise(MADV_DONTFORK).
> + *
> + * The thread receiving the page during the userland page fault
> + * (MADV_USERFAULT) will receive the faulting page in the source vma
> + * through the network, storage or any other I/O device (MADV_DONTFORK
> + * in the source vma avoids remap_anon_pages to fail with -EBUSY if
> + * the process forks before remap_anon_pages is called), then it will
> + * call remap_anon_pages to map the page in the faulting address in
> + * the destination vma.
> + *
> + * This syscall works purely via pagetables, so it's the most
> + * efficient way to move physical non shared anonymous pages across
> + * different virtual addresses. Unlike mremap()/mmap()/munmap() it
> + * does not create any new vmas. The mapping in the destination
> + * address is atomic.
> + *
> + * It only works if the vma protection bits are identical from the
> + * source and destination vma.
> + *
> + * It can remap non shared anonymous pages within the same vma too.
> + *
> + * If the source virtual memory range has any unmapped holes, or if
> + * the destination virtual memory range is not a whole unmapped hole,
> + * remap_anon_pages will fail respectively with -ENOENT or
> + * -EEXIST. This provides a very strict behavior to avoid any chance
> + * of memory corruption going unnoticed if there are userland race
> + * conditions. Only one thread should resolve the userland page fault
> + * at any given time for any given faulting address. This means that
> + * if two threads try to both call remap_anon_pages on the same
> + * destination address at the same time, the second thread will get an
> + * explicit error from this syscall.
> + *
> + * The syscall retval will return "len" is succesful. The syscall
> + * however can be interrupted by fatal signals or errors. If
> + * interrupted it will return the number of bytes successfully
> + * remapped before the interruption if any, or the negative error if
> + * none. It will never return zero. Either it will return an error or
> + * an amount of bytes successfully moved. If the retval reports a
> + * "short" remap, the remap_anon_pages syscall should be repeated by
> + * userland with src+retval, dst+reval, len-retval if it wants to know
> + * about the error that interrupted it.
> + *
> + * The RAP_ALLOW_SRC_HOLES flag can be specified to prevent -ENOENT
> + * errors to materialize if there are holes in the source virtual
> + * range that is being remapped. The holes will be accounted as
> + * successfully remapped in the retval of the syscall. This is mostly
> + * useful to remap hugepage naturally aligned virtual regions without
> + * knowing if there are transparent hugepage in the regions or not,
> + * but preventing the risk of having to split the hugepmd during the
> + * remap.
> + *
> + * If there's any rmap walk that is taking the anon_vma locks without
> + * first obtaining the page lock (for example split_huge_page and
> + * page_referenced_anon), they will have to verify if the
> + * page->mapping has changed after taking the anon_vma lock. If it
> + * changed they should release the lock and retry obtaining a new
> + * anon_vma, because it means the anon_vma was changed by
> + * remap_anon_pages before the lock could be obtained. This is the
> + * only additional complexity added to the rmap code to provide this
> + * anonymous page remapping functionality.
> + */
> +SYSCALL_DEFINE4(remap_anon_pages,
> +               unsigned long, dst_start, unsigned long, src_start,
> +               unsigned long, len, unsigned long, flags)
> +{
> +       struct mm_struct *mm = current->mm;
> +       struct vm_area_struct *src_vma, *dst_vma;
> +       long err = -EINVAL;
> +       pmd_t *src_pmd, *dst_pmd;
> +       pte_t *src_pte, *dst_pte;
> +       spinlock_t *dst_ptl, *src_ptl;
> +       unsigned long src_addr, dst_addr;
> +       int thp_aligned = -1;
> +       long moved = 0;
> +
> +       /*
> +        * Sanitize the syscall parameters:
> +        */
> +       if (src_start & ~PAGE_MASK)
> +               return err;
> +       if (dst_start & ~PAGE_MASK)
> +               return err;
> +       if (len & ~PAGE_MASK)
> +               return err;
> +       if (flags & ~RAP_ALLOW_SRC_HOLES)
> +               return err;
> +
> +       /* Does the address range wrap, or is the span zero-sized? */
> +       if (unlikely(src_start + len <= src_start))
> +               return err;
> +       if (unlikely(dst_start + len <= dst_start))
> +               return err;
> +
> +       down_read(&mm->mmap_sem);
> +
> +       /*
> +        * Make sure the vma is not shared, that the src and dst remap
> +        * ranges are both valid and fully within a single existing
> +        * vma.
> +        */
> +       src_vma = find_vma(mm, src_start);
> +       if (!src_vma || (src_vma->vm_flags & VM_SHARED))
> +               goto out;
> +       if (src_start < src_vma->vm_start ||
> +           src_start + len > src_vma->vm_end)
> +               goto out;
> +
> +       dst_vma = find_vma(mm, dst_start);
> +       if (!dst_vma || (dst_vma->vm_flags & VM_SHARED))
> +               goto out;
> +       if (dst_start < dst_vma->vm_start ||
> +           dst_start + len > dst_vma->vm_end)
> +               goto out;
> +
> +       if (pgprot_val(src_vma->vm_page_prot) !=
> +           pgprot_val(dst_vma->vm_page_prot))
> +               goto out;
> +
> +       /* only allow remapping if both are mlocked or both aren't */
> +       if ((src_vma->vm_flags & VM_LOCKED) ^ (dst_vma->vm_flags & VM_LOCKED))
> +               goto out;
> +
> +       /*
> +        * Ensure the dst_vma has a anon_vma or this page
> +        * would get a NULL anon_vma when moved in the
> +        * dst_vma.
> +        */
> +       err = -ENOMEM;
> +       if (unlikely(anon_vma_prepare(dst_vma)))
> +               goto out;
> +
> +       for (src_addr = src_start, dst_addr = dst_start;
> +            src_addr < src_start + len; ) {
> +               spinlock_t *ptl;
> +               pmd_t dst_pmdval;
> +               BUG_ON(dst_addr >= dst_start + len);
> +               src_pmd = mm_find_pmd(mm, src_addr);
> +               if (unlikely(!src_pmd)) {
> +                       if (!(flags & RAP_ALLOW_SRC_HOLES)) {
> +                               err = -ENOENT;
> +                               break;
> +                       } else {
> +                               src_pmd = mm_alloc_pmd(mm, src_addr);
> +                               if (unlikely(!src_pmd)) {
> +                                       err = -ENOMEM;
> +                                       break;
> +                               }
> +                       }
> +               }
> +               dst_pmd = mm_alloc_pmd(mm, dst_addr);
> +               if (unlikely(!dst_pmd)) {
> +                       err = -ENOMEM;
> +                       break;
> +               }
> +
> +               dst_pmdval = pmd_read_atomic(dst_pmd);
> +               /*
> +                * If the dst_pmd is mapped as THP don't
> +                * override it and just be strict.
> +                */
> +               if (unlikely(pmd_trans_huge(dst_pmdval))) {
> +                       err = -EEXIST;
> +                       break;
> +               }
> +               if (pmd_trans_huge_lock(src_pmd, src_vma, &ptl) == 1) {
> +                       /*
> +                        * Check if we can move the pmd without
> +                        * splitting it. First check the address
> +                        * alignment to be the same in src/dst.  These
> +                        * checks don't actually need the PT lock but
> +                        * it's good to do it here to optimize this
> +                        * block away at build time if
> +                        * CONFIG_TRANSPARENT_HUGEPAGE is not set.
> +                        */
> +                       if (thp_aligned == -1)
> +                               thp_aligned = ((src_addr & ~HPAGE_PMD_MASK) ==
> +                                              (dst_addr & ~HPAGE_PMD_MASK));
> +                       if (!thp_aligned || (src_addr & ~HPAGE_PMD_MASK) ||
> +                           !pmd_none(dst_pmdval) ||
> +                           src_start + len - src_addr < HPAGE_PMD_SIZE) {
> +                               spin_unlock(ptl);
> +                               /* Fall through */
> +                               split_huge_page_pmd(src_vma, src_addr,
> +                                                   src_pmd);
> +                       } else {
> +                               BUG_ON(dst_addr & ~HPAGE_PMD_MASK);
> +                               err = remap_anon_pages_huge_pmd(mm,
> +                                                               dst_pmd,
> +                                                               src_pmd,
> +                                                               dst_pmdval,
> +                                                               dst_vma,
> +                                                               src_vma,
> +                                                               dst_addr,
> +                                                               src_addr);
> +                               cond_resched();
> +
> +                               if (!err) {
> +                                       dst_addr += HPAGE_PMD_SIZE;
> +                                       src_addr += HPAGE_PMD_SIZE;
> +                                       moved += HPAGE_PMD_SIZE;
> +                               }
> +
> +                               if ((!err || err == -EAGAIN) &&
> +                                   fatal_signal_pending(current))
> +                                       err = -EINTR;
> +
> +                               if (err && err != -EAGAIN)
> +                                       break;
> +
> +                               continue;
> +                       }
> +               }
> +
> +               if (pmd_none(*src_pmd)) {
> +                       if (!(flags & RAP_ALLOW_SRC_HOLES)) {
> +                               err = -ENOENT;
> +                               break;
> +                       } else {
> +                               if (unlikely(__pte_alloc(mm, src_vma, src_pmd,
> +                                                        src_addr))) {
> +                                       err = -ENOMEM;
> +                                       break;
> +                               }
> +                       }
> +               }
> +
> +               /*
> +                * We held the mmap_sem for reading so MADV_DONTNEED
> +                * can zap transparent huge pages under us, or the
> +                * transparent huge page fault can establish new
> +                * transparent huge pages under us.
> +                */
> +               if (unlikely(pmd_trans_unstable(src_pmd))) {
> +                       err = -EFAULT;
> +                       break;
> +               }
> +
> +               if (unlikely(pmd_none(dst_pmdval)) &&
> +                   unlikely(__pte_alloc(mm, dst_vma, dst_pmd,
> +                                        dst_addr))) {
> +                       err = -ENOMEM;
> +                       break;
> +               }
> +               /* If an huge pmd materialized from under us fail */
> +               if (unlikely(pmd_trans_huge(*dst_pmd))) {
> +                       err = -EFAULT;
> +                       break;
> +               }
> +
> +               BUG_ON(pmd_none(*dst_pmd));
> +               BUG_ON(pmd_none(*src_pmd));
> +               BUG_ON(pmd_trans_huge(*dst_pmd));
> +               BUG_ON(pmd_trans_huge(*src_pmd));
> +
> +               dst_pte = pte_offset_map(dst_pmd, dst_addr);
> +               src_pte = pte_offset_map(src_pmd, src_addr);
> +               dst_ptl = pte_lockptr(mm, dst_pmd);
> +               src_ptl = pte_lockptr(mm, src_pmd);
> +
> +               err = remap_anon_pages_pte(mm,
> +                                          dst_pte, src_pte, src_pmd,
> +                                          dst_vma, src_vma,
> +                                          dst_addr, src_addr,
> +                                          dst_ptl, src_ptl, flags);
> +
> +               pte_unmap(dst_pte);
> +               pte_unmap(src_pte);
> +               cond_resched();
> +
> +               if (!err) {
> +                       dst_addr += PAGE_SIZE;
> +                       src_addr += PAGE_SIZE;
> +                       moved += PAGE_SIZE;
> +               }
> +
> +               if ((!err || err == -EAGAIN) &&
> +                   fatal_signal_pending(current))
> +                       err = -EINTR;
> +
> +               if (err && err != -EAGAIN)
> +                       break;
> +       }
> +
> +out:
> +       up_read(&mm->mmap_sem);
> +       BUG_ON(moved < 0);
> +       BUG_ON(err > 0);
> +       BUG_ON(!moved && !err);
> +       return moved ? moved : err;
> +}
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 94c37ca..e24cd7c 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -1541,6 +1541,116 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
>  }
>
>  /*
> + * The PT lock for src_pmd and the mmap_sem for reading are held by
> + * the caller, but it must return after releasing the
> + * page_table_lock. We're guaranteed the src_pmd is a pmd_trans_huge
> + * until the PT lock of the src_pmd is released. Just move the page
> + * from src_pmd to dst_pmd if possible. Return zero if succeeded in
> + * moving the page, -EAGAIN if it needs to be repeated by the caller,
> + * or other errors in case of failure.
> + */
> +int remap_anon_pages_huge_pmd(struct mm_struct *mm,
> +                             pmd_t *dst_pmd, pmd_t *src_pmd,
> +                             pmd_t dst_pmdval,
> +                             struct vm_area_struct *dst_vma,
> +                             struct vm_area_struct *src_vma,
> +                             unsigned long dst_addr,
> +                             unsigned long src_addr)
> +{
> +       pmd_t _dst_pmd, src_pmdval;
> +       struct page *src_page;
> +       struct anon_vma *src_anon_vma, *dst_anon_vma;
> +       spinlock_t *src_ptl, *dst_ptl;
> +       pgtable_t pgtable;
> +
> +       src_pmdval = *src_pmd;
> +       src_ptl = pmd_lockptr(mm, src_pmd);
> +
> +       BUG_ON(!pmd_trans_huge(src_pmdval));
> +       BUG_ON(pmd_trans_splitting(src_pmdval));
> +       BUG_ON(!pmd_none(dst_pmdval));
> +       BUG_ON(!spin_is_locked(src_ptl));
> +       BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
> +
> +       src_page = pmd_page(src_pmdval);
> +       BUG_ON(!PageHead(src_page));
> +       BUG_ON(!PageAnon(src_page));
> +       if (unlikely(page_mapcount(src_page) != 1)) {
> +               spin_unlock(src_ptl);
> +               return -EBUSY;
> +       }
> +
> +       get_page(src_page);
> +       spin_unlock(src_ptl);
> +
> +       mmu_notifier_invalidate_range_start(mm, src_addr,
> +                                           src_addr + HPAGE_PMD_SIZE);
> +
> +       /* block all concurrent rmap walks */
> +       lock_page(src_page);
> +
> +       /*
> +        * split_huge_page walks the anon_vma chain without the page
> +        * lock. Serialize against it with the anon_vma lock, the page
> +        * lock is not enough.
> +        */
> +       src_anon_vma = page_get_anon_vma(src_page);
> +       if (!src_anon_vma) {
> +               unlock_page(src_page);
> +               put_page(src_page);
> +               mmu_notifier_invalidate_range_end(mm, src_addr,
> +                                                 src_addr + HPAGE_PMD_SIZE);
> +               return -EAGAIN;
> +       }
> +       anon_vma_lock_write(src_anon_vma);
> +
> +       dst_ptl = pmd_lockptr(mm, dst_pmd);
> +       double_pt_lock(src_ptl, dst_ptl);
> +       if (unlikely(!pmd_same(*src_pmd, src_pmdval) ||
> +                    !pmd_same(*dst_pmd, dst_pmdval) ||
> +                    page_mapcount(src_page) != 1)) {
> +               double_pt_unlock(src_ptl, dst_ptl);
> +               anon_vma_unlock_write(src_anon_vma);
> +               put_anon_vma(src_anon_vma);
> +               unlock_page(src_page);
> +               put_page(src_page);
> +               mmu_notifier_invalidate_range_end(mm, src_addr,
> +                                                 src_addr + HPAGE_PMD_SIZE);
> +               return -EAGAIN;
> +       }
> +
> +       BUG_ON(!PageHead(src_page));
> +       BUG_ON(!PageAnon(src_page));
> +       /* the PT lock is enough to keep the page pinned now */
> +       put_page(src_page);
> +
> +       dst_anon_vma = (void *) dst_vma->anon_vma + PAGE_MAPPING_ANON;
> +       ACCESS_ONCE(src_page->mapping) = (struct address_space *) dst_anon_vma;
> +       ACCESS_ONCE(src_page->index) = linear_page_index(dst_vma, dst_addr);
> +
> +       if (!pmd_same(pmdp_clear_flush(src_vma, src_addr, src_pmd),
> +                     src_pmdval))
> +               BUG();
> +       _dst_pmd = mk_huge_pmd(src_page, dst_vma->vm_page_prot);
> +       _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
> +       set_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);
> +
> +       pgtable = pgtable_trans_huge_withdraw(mm, src_pmd);
> +       pgtable_trans_huge_deposit(mm, dst_pmd, pgtable);
> +       double_pt_unlock(src_ptl, dst_ptl);
> +
> +       anon_vma_unlock_write(src_anon_vma);
> +       put_anon_vma(src_anon_vma);
> +
> +       /* unblock rmap walks */
> +       unlock_page(src_page);
> +
> +       mmu_notifier_invalidate_range_end(mm, src_addr,
> +                                         src_addr + HPAGE_PMD_SIZE);
> +       return 0;
> +}
> +
> +/*
>   * Returns 1 if a given pmd maps a stable (not under splitting) thp.
>   * Returns -1 if it maps a thp under splitting. Returns 0 otherwise.
>   *
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>



-- 
Michael Kerrisk Linux man-pages maintainer;
http://www.kernel.org/doc/man-pages/
Author of "The Linux Programming Interface", http://blog.man7.org/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>




[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]