On Thu, Feb 4, 2021 at 10:34 AM Axel Rasmussen <axelrasmussen@xxxxxxxxxx> wrote: > > Changelog > ========= > > v3->v4: > - Reordered if() branches in hugetlb_mcopy_atomic_pte, so the conditions are > simpler and easier to read. > - Reverted most of the mfill_atomic_pte change (the anon / shmem path). Just > return -EINVAL for CONTINUE, and set zeropage = (mode == > MCOPY_ATOMIC_ZEROPAGE), so we can keep the delta small. > - Split out adding #ifdef CONFIG_USERFAULTFD to a separate patch (instead of > lumping it together with adding UFFDIO_CONTINUE). Also, extended it to make > the same change for shmem as well as suggested by Hugh Dickins. > - Fixed signature of hugetlb_mcopy_atomic_pte for !CONFIG_HUGETLB_PAGE > (signature must be the same in either case). > - Rebased onto a newer version of Peter's patches to disable huge PMD sharing. Apologies for the email noise, of course immediately after doing git send-email I realized I forgot to document one item here: - Relaxed restriction for minor registration to allow any hugetlb VMAs, not just those with VM_SHARED. Fixed setting VM_WRITE flag in a CONTINUE ioctl for non-VM_SHARED VMAs. > > v2->v3: > - Added #ifdef CONFIG_USERFAULTFD around hugetlb helper functions, to fix build > errors when building without CONFIG_USERFAULTFD set. > > v1->v2: > - Fixed a bug in the hugetlb_mcopy_atomic_pte retry case. We now plumb in the > enum mcopy_atomic_mode, so we can differentiate between the three cases this > function needs to handle: > 1) We're doing a COPY op, and need to allocate a page, add to cache, etc. > 2) We're doing a COPY op, but allocation in this function failed previously; > we're in the retry path. The page was allocated, but not e.g. added to page > cache, so that still needs to be done. > 3) We're doing a CONTINUE op, we need to look up an existing page instead of > allocating a new one. > - Rebased onto a newer version of Peter's patches to disable huge PMD sharing, > which fixes syzbot complaints on some non-x86 architectures. > - Moved __VM_UFFD_FLAGS into userfaultfd_k.h, so inline helpers can use it. > - Renamed UFFD_FEATURE_MINOR_FAULT_HUGETLBFS to UFFD_FEATURE_MINOR_HUGETLBFS, > for consistency with other existing feature flags. > - Moved the userfaultfd_minor hook in hugetlb.c into the else block, so we don't > have to explicitly check for !new_page. > > RFC->v1: > - Rebased onto Peter Xu's patches for disabling huge PMD sharing for certain > userfaultfd-registered areas. > - Added commits which update documentation, and add a self test which exercises > the new feature. > - Fixed reporting CONTINUE as a supported ioctl even for non-MINOR ranges. > > Overview > ======== > > This series adds a new userfaultfd registration mode, > UFFDIO_REGISTER_MODE_MINOR. This allows userspace to intercept "minor" faults. > By "minor" fault, I mean the following situation: > > Let there exist two mappings (i.e., VMAs) to the same page(s) (shared memory). > One of the mappings is registered with userfaultfd (in minor mode), and the > other is not. Via the non-UFFD mapping, the underlying pages have already been > allocated & filled with some contents. The UFFD mapping has not yet been > faulted in; when it is touched for the first time, this results in what I'm > calling a "minor" fault. As a concrete example, when working with hugetlbfs, we > have huge_pte_none(), but find_lock_page() finds an existing page. > > We also add a new ioctl to resolve such faults: UFFDIO_CONTINUE. The idea is, > userspace resolves the fault by either a) doing nothing if the contents are > already correct, or b) updating the underlying contents using the second, > non-UFFD mapping (via memcpy/memset or similar, or something fancier like RDMA, > or etc...). In either case, userspace issues UFFDIO_CONTINUE to tell the kernel > "I have ensured the page contents are correct, carry on setting up the mapping". > > Use Case > ======== > > Consider the use case of VM live migration (e.g. under QEMU/KVM): > > 1. While a VM is still running, we copy the contents of its memory to a > target machine. The pages are populated on the target by writing to the > non-UFFD mapping, using the setup described above. The VM is still running > (and therefore its memory is likely changing), so this may be repeated > several times, until we decide the target is "up to date enough". > > 2. We pause the VM on the source, and start executing on the target machine. > During this gap, the VM's user(s) will *see* a pause, so it is desirable to > minimize this window. > > 3. Between the last time any page was copied from the source to the target, and > when the VM was paused, the contents of that page may have changed - and > therefore the copy we have on the target machine is out of date. Although we > can keep track of which pages are out of date, for VMs with large amounts of > memory, it is "slow" to transfer this information to the target machine. We > want to resume execution before such a transfer would complete. > > 4. So, the guest begins executing on the target machine. The first time it > touches its memory (via the UFFD-registered mapping), userspace wants to > intercept this fault. Userspace checks whether or not the page is up to date, > and if not, copies the updated page from the source machine, via the non-UFFD > mapping. Finally, whether a copy was performed or not, userspace issues a > UFFDIO_CONTINUE ioctl to tell the kernel "I have ensured the page contents > are correct, carry on setting up the mapping". > > We don't have to do all of the final updates on-demand. The userfaultfd manager > can, in the background, also copy over updated pages once it receives the map of > which pages are up-to-date or not. > > Interaction with Existing APIs > ============================== > > Because it's possible to combine registration modes (e.g. a single VMA can be > userfaultfd-registered MINOR | MISSING), and because it's up to userspace how to > resolve faults once they are received, I spent some time thinking through how > the existing API interacts with the new feature. > > UFFDIO_CONTINUE cannot be used to resolve non-minor faults, as it does not > allocate a new page. If UFFDIO_CONTINUE is used on a non-minor fault: > > - For non-shared memory or shmem, -EINVAL is returned. > - For hugetlb, -EFAULT is returned. > > UFFDIO_COPY and UFFDIO_ZEROPAGE cannot be used to resolve minor faults. Without > modifications, the existing codepath assumes a new page needs to be allocated. > This is okay, since userspace must have a second non-UFFD-registered mapping > anyway, thus there isn't much reason to want to use these in any case (just > memcpy or memset or similar). > > - If UFFDIO_COPY is used on a minor fault, -EEXIST is returned. > - If UFFDIO_ZEROPAGE is used on a minor fault, -EEXIST is returned (or -EINVAL > in the case of hugetlb, as UFFDIO_ZEROPAGE is unsupported in any case). > - UFFDIO_WRITEPROTECT simply doesn't work with shared memory, and returns > -ENOENT in that case (regardless of the kind of fault). > > Dependencies > ============ > > I've included 4 commits from Peter Xu's larger series > (https://lore.kernel.org/patchwork/cover/1366017/) in this series. My changes > depend on his work, to disable huge PMD sharing for MINOR registered userfaultfd > areas. I included the 4 commits directly because a) it lets this series just be > applied and work as-is, and b) they are fairly standalone, and could potentially > be merged even without the rest of the larger series Peter submitted. Thanks > Peter! > > Also, although it doesn't affect minor fault handling, I did notice that the > userfaultfd self test sometimes experienced memory corruption > (https://lore.kernel.org/patchwork/cover/1356755/). For anyone testing this > series, it may be useful to apply that series first to fix the selftest > flakiness. That series doesn't have to be merged into mainline / maintaner > branches before mine, though. > > Future Work > =========== > > Currently the patchset only supports hugetlbfs. There is no reason it can't work > with shmem, but I expect hugetlbfs to be much more commonly used since we're > talking about backing guest memory for VMs. I plan to implement shmem support in > a follow-up patch series. > > Axel Rasmussen (6): > userfaultfd: add minor fault registration mode > userfaultfd: disable huge PMD sharing for MINOR registered VMAs > userfaultfd: hugetlbfs: only compile UFFD helpers if config enabled > userfaultfd: add UFFDIO_CONTINUE ioctl > userfaultfd: update documentation to describe minor fault handling > userfaultfd/selftests: add test exercising minor fault handling > > Peter Xu (4): > hugetlb: Pass vma into huge_pte_alloc() and huge_pmd_share() > hugetlb/userfaultfd: Forbid huge pmd sharing when uffd enabled > mm/hugetlb: Move flush_hugetlb_tlb_range() into hugetlb.h > hugetlb/userfaultfd: Unshare all pmds for hugetlbfs when register wp > > Documentation/admin-guide/mm/userfaultfd.rst | 107 ++++++---- > arch/arm64/mm/hugetlbpage.c | 7 +- > arch/ia64/mm/hugetlbpage.c | 3 +- > arch/mips/mm/hugetlbpage.c | 4 +- > arch/parisc/mm/hugetlbpage.c | 2 +- > arch/powerpc/mm/hugetlbpage.c | 3 +- > arch/s390/mm/hugetlbpage.c | 2 +- > arch/sh/mm/hugetlbpage.c | 2 +- > arch/sparc/mm/hugetlbpage.c | 6 +- > fs/proc/task_mmu.c | 1 + > fs/userfaultfd.c | 196 ++++++++++++++++--- > include/linux/hugetlb.h | 22 ++- > include/linux/mm.h | 1 + > include/linux/mmu_notifier.h | 1 + > include/linux/userfaultfd_k.h | 49 ++++- > include/trace/events/mmflags.h | 1 + > include/uapi/linux/userfaultfd.h | 36 +++- > mm/hugetlb.c | 113 +++++++---- > mm/userfaultfd.c | 51 +++-- > tools/testing/selftests/vm/userfaultfd.c | 147 +++++++++++++- > 20 files changed, 601 insertions(+), 153 deletions(-) > > -- > 2.30.0.365.g02bc693789-goog >
