On Mon, May 15 2023 at 21:46, Thomas Gleixner wrote:
> On Mon, May 15 2023 at 17:59, Russell King wrote:
>> On Mon, May 15, 2023 at 06:43:40PM +0200, Thomas Gleixner wrote:
> That reproduces in a VM easily and has exactly the same behaviour:
>
> Extra page[s] via The actual allocation
> _vm_unmap_aliases() Pages Pages Flush start Pages
> alloc: ffffc9000058e000 2
> free : ffff888144751000 1 ffffc9000058e000 2 ffff888144751000 17312759359
>
> alloc: ffffc90000595000 2
> free : ffff8881424f0000 1 ffffc90000595000 2 ffff8881424f0000 17312768167
>
> .....
>
> seccomp seems to install 29 BPF programs for that process. So on exit()
> this results in 29 full TLB flushes on x86, where each of them is used
> to flush exactly three TLB entries.
>
> The actual two page allocation (ffffc9...) is in the vmalloc space, the
> extra page (ffff88...) is in the direct mapping.
I tried to flush them one by one, which is actually slightly slower.
That's not surprising as there are 3 * 29 instead of 29 IPIs and the
IPIs dominate the picture.
But that's not necessarily true for ARM32 as there are no IPIs involved
on the machine we are using, which is a dual-core Cortex-A9.
So I came up with the hack below, which is equally fast as the full
flush variant while the performance impact on the other CPUs is minimally
lower according to perf.
That probably should have another argument which tells how many TLBs
this flush affects, i.e. 3 in this example, so an architecture can
sensibly decide whether it wants to use flush all or not.
Thanks,
tglx
---
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1728,6 +1728,7 @@ static bool __purge_vmap_area_lazy(unsig
unsigned int num_purged_areas = 0;
struct list_head local_purge_list;
struct vmap_area *va, *n_va;
+ struct vmap_area tmp = { .va_start = start, .va_end = end };
lockdep_assert_held(&vmap_purge_lock);
@@ -1747,7 +1748,12 @@ static bool __purge_vmap_area_lazy(unsig
list_last_entry(&local_purge_list,
struct vmap_area, list)->va_end);
- flush_tlb_kernel_range(start, end);
+ if (tmp.va_end > tmp.va_start)
+ list_add(&tmp.list, &local_purge_list);
+ flush_tlb_kernel_vas(&local_purge_list);
+ if (tmp.va_end > tmp.va_start)
+ list_del(&tmp.list);
+
resched_threshold = lazy_max_pages() << 1;
spin_lock(&free_vmap_area_lock);
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -10,6 +10,7 @@
#include <linux/debugfs.h>
#include <linux/sched/smt.h>
#include <linux/task_work.h>
+#include <linux/vmalloc.h>
#include <asm/tlbflush.h>
#include <asm/mmu_context.h>
@@ -1081,6 +1082,24 @@ void flush_tlb_kernel_range(unsigned lon
}
}
+static void do_flush_vas(void *arg)
+{
+ struct list_head *list = arg;
+ struct vmap_area *va;
+ unsigned long addr;
+
+ list_for_each_entry(va, list, list) {
+ /* flush range by one by one 'invlpg' */
+ for (addr = va->va_start; addr < va->va_end; addr += PAGE_SIZE)
+ flush_tlb_one_kernel(addr);
+ }
+}
+
+void flush_tlb_kernel_vas(struct list_head *list)
+{
+ on_each_cpu(do_flush_vas, list, 1);
+}
+
/*
* This can be used from process context to figure out what the value of
* CR3 is without needing to do a (slow) __read_cr3().
--- a/include/linux/vmalloc.h
+++ b/include/linux/vmalloc.h
@@ -295,4 +295,6 @@ bool vmalloc_dump_obj(void *object);
static inline bool vmalloc_dump_obj(void *object) { return false; }
#endif
+void flush_tlb_kernel_vas(struct list_head *list);
+
#endif /* _LINUX_VMALLOC_H */
