On Tue, Feb 25, 2025 at 10:00:40PM -0500, Rik van Riel wrote: > The flush_tlb_all() function is not used a whole lot, but we might > as well use broadcast TLB flushing there, too. > > Signed-off-by: Rik van Riel <riel@xxxxxxxxxxx> > Tested-by: Manali Shukla <Manali.Shukla@xxxxxxx> > Tested-by: Brendan Jackman <jackmanb@xxxxxxxxxx> > Tested-by: Michael Kelley <mhklinux@xxxxxxxxxxx> > --- > arch/x86/mm/tlb.c | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) Edits ontop: --- /tmp/current.patch 2025-02-28 23:18:51.670490799 +0100 +++ /tmp/0001-x86-mm-Use-INVLPGB-in-flush_tlb_all.patch 2025-02-28 23:17:48.590844991 +0100 @@ -1,22 +1,23 @@ +From 5bdf59c0589b71328bd340ea48a00917def62dc0 Mon Sep 17 00:00:00 2001 From: Rik van Riel <riel@xxxxxxxxxxx> Date: Tue, 25 Feb 2025 22:00:40 -0500 -Subject: x86/mm: Use INVLPGB in flush_tlb_all +Subject: [PATCH] x86/mm: Use INVLPGB in flush_tlb_all() -The flush_tlb_all() function is not used a whole lot, but we might -as well use broadcast TLB flushing there, too. +The flush_tlb_all() function is not used a whole lot, but it might as +well use broadcast TLB flushing there, too. + + [ bp: Massage, restore balanced if-else branches in the function, + comment some. ] Signed-off-by: Rik van Riel <riel@xxxxxxxxxxx> Signed-off-by: Borislav Petkov (AMD) <bp@xxxxxxxxx> -Tested-by: Manali Shukla <Manali.Shukla@xxxxxxx> -Tested-by: Brendan Jackman <jackmanb@xxxxxxxxxx> -Tested-by: Michael Kelley <mhklinux@xxxxxxxxxxx> Link: https://lore.kernel.org/r/20250226030129.530345-6-riel@xxxxxxxxxxx --- - arch/x86/mm/tlb.c | 10 +++++++++- - 1 file changed, 9 insertions(+), 1 deletion(-) + arch/x86/mm/tlb.c | 17 +++++++++++++++-- + 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c -index f44a03bca41c..a6cd61d5f423 100644 +index 5c44b94ad5af..f49627e02311 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1064,7 +1064,6 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, @@ -27,22 +28,29 @@ index f44a03bca41c..a6cd61d5f423 100644 static void do_flush_tlb_all(void *info) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED); -@@ -1074,6 +1073,15 @@ static void do_flush_tlb_all(void *info) +@@ -1074,7 +1073,21 @@ static void do_flush_tlb_all(void *info) void flush_tlb_all(void) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); +- on_each_cpu(do_flush_tlb_all, NULL, 1); + + /* First try (faster) hardware-assisted TLB invalidation. */ + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { ++ /* ++ * TLBSYNC at the end needs to make sure all flushes done ++ * on the current CPU have been executed system-wide. ++ * Therefore, make sure nothing gets migrated ++ * in-between but disable preemption as it is cheaper. ++ */ + guard(preempt)(); + invlpgb_flush_all(); -+ return; ++ } else { ++ /* Fall back to the IPI-based invalidation. */ ++ on_each_cpu(do_flush_tlb_all, NULL, 1); + } -+ -+ /* Fall back to the IPI-based invalidation. */ - on_each_cpu(do_flush_tlb_all, NULL, 1); } + /* Flush an arbitrarily large range of memory with INVLPGB. */ -- -- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
