The patch titled
Subject: mm: vmalloc: add va_alloc() helper
has been added to the -mm mm-unstable branch. Its filename is
mm-vmalloc-add-va_alloc-helper.patch
This patch will shortly appear at
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-vmalloc-add-va_alloc-helper.patch
This patch will later appear in the mm-unstable branch at
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Before you just go and hit "reply", please:
a) Consider who else should be cc'ed
b) Prefer to cc a suitable mailing list as well
c) Ideally: find the original patch on the mailing list and do a
reply-to-all to that, adding suitable additional cc's
*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***
The -mm tree is included into linux-next via the mm-everything
branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
and is updated there every 2-3 working days
------------------------------------------------------
From: "Uladzislau Rezki (Sony)" <urezki@xxxxxxxxx>
Subject: mm: vmalloc: add va_alloc() helper
Date: Tue, 29 Aug 2023 10:11:34 +0200
Patch series "Mitigate a vmap lock contention", v2.
Test on AMD Ryzen Threadripper 3970X 32-Core Processor: sudo
./test_vmalloc.sh run_test_mask=127 nr_threads=64
<v6.5-rc6 perf>
94.17% 0.90% [kernel] [k] _raw_spin_lock
93.27% 93.05% [kernel] [k] native_queued_spin_lock_slowpath
74.69% 0.25% [kernel] [k] __vmalloc_node_range
72.64% 0.01% [kernel] [k] __get_vm_area_node
72.04% 0.89% [kernel] [k] alloc_vmap_area
42.17% 0.00% [kernel] [k] vmalloc
32.53% 0.00% [kernel] [k] __vmalloc_node
24.91% 0.25% [kernel] [k] vfree
24.32% 0.01% [kernel] [k] remove_vm_area
22.63% 0.21% [kernel] [k] find_unlink_vmap_area
15.51% 0.00% [unknown] [k] 0xffffffffc09a74ac
14.35% 0.00% [kernel] [k] ret_from_fork_asm
14.35% 0.00% [kernel] [k] ret_from_fork
14.35% 0.00% [kernel] [k] kthread
<v6.5-rc6 perf>
vs
<v6.5-rc6+v2 perf>
74.32% 2.42% [kernel] [k] __vmalloc_node_range
69.58% 0.01% [kernel] [k] vmalloc
54.21% 1.17% [kernel] [k] __alloc_pages_bulk
48.13% 47.91% [kernel] [k] clear_page_orig
43.60% 0.01% [unknown] [k] 0xffffffffc082f16f
32.06% 0.00% [kernel] [k] ret_from_fork_asm
32.06% 0.00% [kernel] [k] ret_from_fork
32.06% 0.00% [kernel] [k] kthread
31.30% 0.00% [unknown] [k] 0xffffffffc082f889
22.98% 4.16% [kernel] [k] vfree
14.36% 0.28% [kernel] [k] __get_vm_area_node
13.43% 3.35% [kernel] [k] alloc_vmap_area
10.86% 0.04% [kernel] [k] remove_vm_area
8.89% 2.75% [kernel] [k] _raw_spin_lock
7.19% 0.00% [unknown] [k] 0xffffffffc082fba3
6.65% 1.37% [kernel] [k] free_unref_page
6.13% 6.11% [kernel] [k] native_queued_spin_lock_slowpath
<v6.5-rc6+v2 perf>
On smaller systems, for example, 8xCPU Hikey960 board the contention is
not that high and is approximately ~16 percent.
This patch (of 9):
Currently __alloc_vmap_area() function contains an open codded logic that
finds and adjusts a VA based on allocation request.
Introduce a va_alloc() helper that adjusts found VA only. It will be used
later at least in two places.
There is no a functional change as a result of this patch.
Link: https://lkml.kernel.org/r/20230829081142.3619-1-urezki@xxxxxxxxx
Link: https://lkml.kernel.org/r/20230829081142.3619-2-urezki@xxxxxxxxx
Signed-off-by: Uladzislau Rezki (Sony) <urezki@xxxxxxxxx>
Reviewed-by: Christoph Hellwig <hch@xxxxxx>
Reviewed-by: Lorenzo Stoakes <lstoakes@xxxxxxxxx>
Cc: Baoquan He <bhe@xxxxxxxxxx>
Cc: Dave Chinner <david@xxxxxxxxxxxxx>
Cc: Joel Fernandes (Google) <joel@xxxxxxxxxxxxxxxxx>
Cc: Liam R. Howlett <Liam.Howlett@xxxxxxxxxx>
Cc: Matthew Wilcox (Oracle) <willy@xxxxxxxxxxxxx>
Cc: Oleksiy Avramchenko <oleksiy.avramchenko@xxxxxxxx>
Cc: Paul E. McKenney <paulmck@xxxxxxxxxx>
Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
---
mm/vmalloc.c | 41 ++++++++++++++++++++++++++++-------------
1 file changed, 28 insertions(+), 13 deletions(-)
--- a/mm/vmalloc.c~mm-vmalloc-add-va_alloc-helper
+++ a/mm/vmalloc.c
@@ -1481,6 +1481,32 @@ adjust_va_to_fit_type(struct rb_root *ro
return 0;
}
+static unsigned long
+va_alloc(struct vmap_area *va,
+ struct rb_root *root, struct list_head *head,
+ unsigned long size, unsigned long align,
+ unsigned long vstart, unsigned long vend)
+{
+ unsigned long nva_start_addr;
+ int ret;
+
+ if (va->va_start > vstart)
+ nva_start_addr = ALIGN(va->va_start, align);
+ else
+ nva_start_addr = ALIGN(vstart, align);
+
+ /* Check the "vend" restriction. */
+ if (nva_start_addr + size > vend)
+ return vend;
+
+ /* Update the free vmap_area. */
+ ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size);
+ if (WARN_ON_ONCE(ret))
+ return vend;
+
+ return nva_start_addr;
+}
+
/*
* Returns a start address of the newly allocated area, if success.
* Otherwise a vend is returned that indicates failure.
@@ -1493,7 +1519,6 @@ __alloc_vmap_area(struct rb_root *root,
bool adjust_search_size = true;
unsigned long nva_start_addr;
struct vmap_area *va;
- int ret;
/*
* Do not adjust when:
@@ -1511,18 +1536,8 @@ __alloc_vmap_area(struct rb_root *root,
if (unlikely(!va))
return vend;
- if (va->va_start > vstart)
- nva_start_addr = ALIGN(va->va_start, align);
- else
- nva_start_addr = ALIGN(vstart, align);
-
- /* Check the "vend" restriction. */
- if (nva_start_addr + size > vend)
- return vend;
-
- /* Update the free vmap_area. */
- ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size);
- if (WARN_ON_ONCE(ret))
+ nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend);
+ if (nva_start_addr == vend)
return vend;
#if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
_
Patches currently in -mm which might be from urezki@xxxxxxxxx are
mm-vmalloc-add-va_alloc-helper.patch
mm-vmalloc-rename-adjust_va_to_fit_type-function.patch
mm-vmalloc-move-vmap_init_free_space-down-in-vmallocc.patch
mm-vmalloc-remove-global-vmap_area_root-rb-tree.patch
mm-vmalloc-remove-global-vmap_area_root-rb-tree-fix.patch
mm-vmalloc-remove-global-purge_vmap_area_root-rb-tree.patch
mm-vmalloc-offload-free_vmap_area_lock-lock.patch
mm-vmalloc-support-multiple-nodes-in-vread_iter.patch
mm-vmalloc-support-multiple-nodes-in-vmallocinfo.patch
mm-vmalloc-set-nr_nodes-node_size-based-on-cpu-cores.patch
