* Peng Zhang <zhangpeng.00@xxxxxxxxxxxxx> [230908 05:59]: > > > 在 2023/9/8 04:14, Liam R. Howlett 写道: > > * Peng Zhang <zhangpeng.00@xxxxxxxxxxxxx> [230830 08:58]: > > > Use __mt_dup() to duplicate the old maple tree in dup_mmap(), and then > > > directly modify the entries of VMAs in the new maple tree, which can > > > get better performance. The optimization effect is proportional to the > > > number of VMAs. > > > > > > There is a "spawn" in byte-unixbench[1], which can be used to test the > > > performance of fork(). I modified it slightly to make it work with > > > different number of VMAs. > > > > > > Below are the test numbers. There are 21 VMAs by default. The first row > > > indicates the number of added VMAs. The following two lines are the > > > number of fork() calls every 10 seconds. These numbers are different > > > from the test results in v1 because this time the benchmark is bound to > > > a CPU. This way the numbers are more stable. > > > > > > Increment of VMAs: 0 100 200 400 800 1600 3200 6400 > > > 6.5.0-next-20230829: 111878 75531 53683 35282 20741 11317 6110 3158 > > > Apply this patchset: 114531 85420 64541 44592 28660 16371 9038 4831 > > > +2.37% +13.09% +20.23% +26.39% +38.18% +44.66% +47.92% +52.98% > > > > Thanks! > > > > Can you include 21 in this table since it's the default? > Maybe I didn't express clearly, "Increment of VMAs" means the number of > VMAs added on the basis of 21 VMAs. Ah, I see. Thanks. > > > > > > > > [1] https://github.com/kdlucas/byte-unixbench/tree/master > > > > > > Signed-off-by: Peng Zhang <zhangpeng.00@xxxxxxxxxxxxx> > > > --- > > > kernel/fork.c | 34 ++++++++++++++++++++++++++-------- > > > mm/mmap.c | 14 ++++++++++++-- > > > 2 files changed, 38 insertions(+), 10 deletions(-) > > > > > > diff --git a/kernel/fork.c b/kernel/fork.c > > > index 3b6d20dfb9a8..e6299adefbd8 100644 > > > --- a/kernel/fork.c > > > +++ b/kernel/fork.c > > > @@ -650,7 +650,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, > > > int retval; > > > unsigned long charge = 0; > > > LIST_HEAD(uf); > > > - VMA_ITERATOR(old_vmi, oldmm, 0); > > > VMA_ITERATOR(vmi, mm, 0); > > > uprobe_start_dup_mmap(); > > > @@ -678,17 +677,39 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, > > > goto out; > > > khugepaged_fork(mm, oldmm); > > > - retval = vma_iter_bulk_alloc(&vmi, oldmm->map_count); > > > - if (retval) > > > + /* Use __mt_dup() to efficiently build an identical maple tree. */ > > > + retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_NOWAIT | __GFP_NOWARN); > > > > Apparently the flags should be GFP_KERNEL here so that compaction can > > run. > OK, I'll change it to GFP_KERNEL. > > > > > + if (unlikely(retval)) > > > goto out; > > > mt_clear_in_rcu(vmi.mas.tree); > > > - for_each_vma(old_vmi, mpnt) { > > > + for_each_vma(vmi, mpnt) { > > > struct file *file; > > > vma_start_write(mpnt); > > > if (mpnt->vm_flags & VM_DONTCOPY) { > > > vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mpnt)); > > > + > > > + /* > > > + * Since the new tree is exactly the same as the old one, > > > + * we need to remove the unneeded VMAs. > > > + */ > > > + mas_store(&vmi.mas, NULL); > > > + > > > + /* > > > + * Even removing an entry may require memory allocation, > > > + * and if removal fails, we use XA_ZERO_ENTRY to mark > > > + * from which VMA it failed. The case of encountering > > > + * XA_ZERO_ENTRY will be handled in exit_mmap(). > > > + */ > > > + if (unlikely(mas_is_err(&vmi.mas))) { > > > + retval = xa_err(vmi.mas.node); > > > + mas_reset(&vmi.mas); > > > + if (mas_find(&vmi.mas, ULONG_MAX)) > > > + mas_store(&vmi.mas, XA_ZERO_ENTRY); > > > + goto loop_out; > > > + } > > > + > > > > Storing NULL may need extra space as you noted, so we need to be careful > > what happens if we don't have that space. We should have a testcase to > > test this scenario. > > > > mas_store_gfp() should be used with GFP_KERNEL. The VMAs use GFP_KERNEL > > in this function, see vm_area_dup(). > > > > Don't use the exit_mmap() path to undo a failed fork. You've added > > checks and complications to the exit path for all tasks in the very > > unlikely event that we run out of memory when we hit a very unlikely > > VM_DONTCOPY flag. > > > > I see the issue with having a portion of the tree with new VMAs that are > > accounted and a portion of the tree that has old VMAs that should not be > > looked at. It was clever to use the XA_ZERO_ENTRY as a stop point, but > > we cannot add that complication to the exit path and then there is the > > OOM race to worry about (maybe, I am not sure since this MM isn't > > active yet). > > > > Using what is done in exit_mmap() and do_vmi_align_munmap() as a > > prototype, we can do something like the *untested* code below: > > > > if (unlikely(mas_is_err(&vmi.mas))) { > > unsigned long max = vmi.index; > > > > retval = xa_err(vmi.mas.node); > > mas_set(&vmi.mas, 0); > > tmp = mas_find(&vmi.mas, ULONG_MAX); > > if (tmp) { /* Not the first VMA failed */ > > unsigned long nr_accounted = 0; > > > > unmap_region(mm, &vmi.mas, vma, NULL, mpnt, 0, max, max, > > true); > > do { > > if (vma->vm_flags & VM_ACCOUNT) > > nr_accounted += vma_pages(vma); > > remove_vma(vma, true); > > cond_resched(); > > vma = mas_find(&vmi.mas, max - 1); > > } while (vma != NULL); > > > > vm_unacct_memory(nr_accounted); > > } > > __mt_destroy(&mm->mm_mt); > > goto loop_out; > > } > > > > Once exit_mmap() is called, the check for OOM (no vma) will catch that > > nothing is left to do. > > > > It might be worth making an inline function to do this to keep the fork > > code clean. We should test this by detecting a specific task name and > > returning a failure at a given interval: > > > > if (!strcmp(current->comm, "fork_test") { > > ... > > } > > Thank you for your suggestion, I will do this in the next version. > > > > > > > continue; > > > } > > > charge = 0; > > > @@ -750,8 +771,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, > > > hugetlb_dup_vma_private(tmp); > > > /* Link the vma into the MT */ > > > - if (vma_iter_bulk_store(&vmi, tmp)) > > > - goto fail_nomem_vmi_store; > > > + mas_store(&vmi.mas, tmp); > > > mm->map_count++; > > > if (!(tmp->vm_flags & VM_WIPEONFORK)) > > > @@ -778,8 +798,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, > > > uprobe_end_dup_mmap(); > > > return retval; > > > -fail_nomem_vmi_store: > > > - unlink_anon_vmas(tmp); > > > fail_nomem_anon_vma_fork: > > > mpol_put(vma_policy(tmp)); > > > fail_nomem_policy: > > > diff --git a/mm/mmap.c b/mm/mmap.c > > > index b56a7f0c9f85..dfc6881be81c 100644 > > > --- a/mm/mmap.c > > > +++ b/mm/mmap.c > > > @@ -3196,7 +3196,11 @@ void exit_mmap(struct mm_struct *mm) > > > arch_exit_mmap(mm); > > > vma = mas_find(&mas, ULONG_MAX); > > > - if (!vma) { > > > + /* > > > + * If dup_mmap() fails to remove a VMA marked VM_DONTCOPY, > > > + * xa_is_zero(vma) may be true. > > > + */ > > > + if (!vma || xa_is_zero(vma)) { > > > /* Can happen if dup_mmap() received an OOM */ > > > mmap_read_unlock(mm); > > > return; > > > @@ -3234,7 +3238,13 @@ void exit_mmap(struct mm_struct *mm) > > > remove_vma(vma, true); > > > count++; > > > cond_resched(); > > > - } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); > > > + vma = mas_find(&mas, ULONG_MAX); > > > + /* > > > + * If xa_is_zero(vma) is true, it means that subsequent VMAs > > > + * donot need to be removed. Can happen if dup_mmap() fails to > > > + * remove a VMA marked VM_DONTCOPY. > > > + */ > > > + } while (vma != NULL && !xa_is_zero(vma)); > > > BUG_ON(count != mm->map_count); > > > -- > > > 2.20.1 > > > > >
