This is a series of patches that try to improve merge success rate when VMAs are being moved, resized or otherwise modified. Motivation In the current kernel it is impossible to merge two anonymous VMAs if one of them was moved. That is because VMA's page offset is set according to the virtual address where it was created and in order to merge two VMAs page offsets need to follow up. Another problem when merging two faulted VMA's is their anon_vma. In current kernel these anon_vmas have to be the one and the same. Otherwise merge is again not allowed. There are several places from which vma_merge() is called and therefore several use cases that might profit from this upgrade. These include mmap (that fills a hole between two VMAs), mremap (that moves VMA next to another one or again perfectly fills a hole), mprotect (that modifies protection and allows merging with a neighbor) and brk (that expands VMA so that it is adjacent to a neighbor). Missed merge opportunities increase the number of VMAs of a process and in some cases can cause problems when a max count is reached. Solution The series solves the first problem with page offsets by updating them when the VMA is moved to a different virtual address (patch 4). As for the second problem, merging of VMAs with different anon_vma is allowed under some conditions (patch 5). Another missed opportunity in the current kernel is when mremap enlarges an already existing VMA and it is possible to merge with following VMA (patch 2). Patch 1 refactors function vma_merge and makes it easier to understand and also allows relatively seamless tracing of successful merges introduced by the patch 6. Patch 3 introduces migration waiting and rmap locking into the pagewalk mechnism, which is necessary for patche 4 and 5. Limitations For both problems solution works only for VMAs that do not share physical pages with other processes (usually child or parent processes). This is checked by looking at anon_vma of the respective VMA and also by looking at mapcount of individual pages. The reason why it is not possible or at least not easy to accomplish is that each physical page has a pointer to anon_vma and page offset. And when this physical page is shared we cannot simply change these parameters without affecting all of the VMAs mapping this physical page. Good thing is that this case amounts only for about 1-3% of all merges (measured on jemalloc (0%), redis (2.7%) and kcbench (1.2%) tests) that fail to merge in the current kernel. Measuring also shows slight increase in running time, jemalloc (0.3%), redis (1%), kcbench (1%). More extensive data can be viewed at https://home.alabanda.cz/share/results.png Changelog Pagewalk - previously page struct has been accessed using follow_page() which goes through the whole pagewalk for each call. This version uses walk_page_vma() which goes through all the necessary pages at the pte level (vm_normal_page() is used there). Pgoff update was previously performed at the beginning of copy_vma() for all the pages (page->index specifically) and also for the pgoff variable used to construct the VMA copy. Now the update of individual pages is done later in move_page_tables(). This makes more sense because move_page_tables() moves all the pages to the new VMA anyway and this again spares some otherwise duplicate page walking. Anon_vma update for mprotect cases is done in __vma_adjust(). For mremap cases the update is done in move_page_tables() together with the page offset update. Previously the anon_vma update was always handled in __vma_adjust() but it was not done in all necessary cases. More details are mentioned in the concerned patches. Questions Is it necessary to check mapcount of individual pages of the VMA to determine if they are shared with other processes? Is it even possible when VMA or respectivelly its anon_vma is not shared? So far as my knowledge of kernel goes, it seems that checking individual pages is not necessary and check on the level of anon_vma is suficient. KSM would theoretically interfere with page mapcount, but it is temporarily disabled before move_vma() in mremap syscall happens. Does anyone know about something else that can change mapcount without anon_vma knowing? This series of patches and documentation of the related code will be part of my master's thesis. This patch series is based on tag v5.18-rc2. This is a third version. Jakub Matěna (6): mm: refactor of vma_merge() mm: add merging after mremap resize mm: add migration waiting and rmap locking to pagewalk mm: adjust page offset in mremap mm: enable merging of VMAs with different anon_vmas mm: add tracing for VMA merges fs/exec.c | 2 +- fs/proc/task_mmu.c | 4 +- include/linux/mm.h | 4 +- include/linux/pagewalk.h | 15 +- include/linux/rmap.h | 19 ++- include/trace/events/mmap.h | 83 ++++++++++ mm/internal.h | 12 ++ mm/mmap.c | 291 ++++++++++++++++++++++++++---------- mm/mremap.c | 153 ++++++++++++++----- mm/pagewalk.c | 75 +++++++++- mm/rmap.c | 144 ++++++++++++++++++ 11 files changed, 670 insertions(+), 132 deletions(-) -- 2.35.1
