From: Li Zhe <lizhe.67@xxxxxxxxxxxxx>
In function collapse_huge_page(), we drop mmap read lock and get
mmap write lock to prevent most accesses to pagetables. There is
a small time window to allow other tasks to acquire the mmap lock.
With the use of upgrade_read(), we don't need to check vma and pmd
again in most cases.
Signed-off-by: Li Zhe <lizhe.67@xxxxxxxxxxxxx>
---
mm/khugepaged.c | 36 +++++++++++++++++++-----------------
1 file changed, 19 insertions(+), 17 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f9c39898eaff..934051274f7a 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1142,23 +1142,25 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
goto out_nolock;
}
- mmap_read_unlock(mm);
- /*
- * Prevent all access to pagetables with the exception of
- * gup_fast later handled by the ptep_clear_flush and the VM
- * handled by the anon_vma lock + PG_lock.
- *
- * UFFDIO_MOVE is prevented to race as well thanks to the
- * mmap_lock.
- */
- mmap_write_lock(mm);
- result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
- if (result != SCAN_SUCCEED)
- goto out_up_write;
- /* check if the pmd is still valid */
- result = check_pmd_still_valid(mm, address, pmd);
- if (result != SCAN_SUCCEED)
- goto out_up_write;
+ if (upgrade_read(&mm->mmap_lock)) {
+ mmap_read_unlock(mm);
+ /*
+ * Prevent all access to pagetables with the exception of
+ * gup_fast later handled by the ptep_clear_flush and the VM
+ * handled by the anon_vma lock + PG_lock.
+ *
+ * UFFDIO_MOVE is prevented to race as well thanks to the
+ * mmap_lock.
+ */
+ mmap_write_lock(mm);
+ result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
+ if (result != SCAN_SUCCEED)
+ goto out_up_write;
+ /* check if the pmd is still valid */
+ result = check_pmd_still_valid(mm, address, pmd);
+ if (result != SCAN_SUCCEED)
+ goto out_up_write;
+ }
vma_start_write(vma);
anon_vma_lock_write(vma->anon_vma);
--
2.20.1
