From: Takao Indoh <indou.takao@xxxxxxxxxxx>
I found a performance issue related on the implementation of Linux's TLB
flush for arm64.
When I run a single-threaded test program on moderate environment, it
usually takes 39ms to finish its work. However, when I put a small
apprication, which just calls mprotest() continuously, on one of sibling
cores and run it simultaneously, the test program slows down significantly.
It becomes 49ms(125%) on ThunderX2. I also detected the same problem on
ThunderX1 and Fujitsu A64FX.
I suppose the root cause of this issue is the implementation of Linux's TLB
flush for arm64, especially use of TLBI-is instruction which is a broadcast
to all processor core on the system. In case of the above situation,
TLBI-is is called by mprotect().
This is not a problem for small environment, but this causes a significant
performance noise for large-scale HPC environment, which has more than
thousand nodes with low latency interconnect.
To fix this problem, this patch adds new boot parameter
'disable_tlbflush_is'. In the case of flush_tlb_mm() *without* this
parameter, TLB entry is invalidated by __tlbi(aside1is, asid). By this
instruction, all CPUs within the same inner shareable domain check if there
are TLB entries which have this ASID, this causes performance noise. OTOH,
when this new parameter is specified, TLB entry is invalidated by
__tlbi(aside1, asid) only on the CPUs specified by mm_cpumask(mm).
Therefore TLB flush is done on minimal CPUs and performance problem does
not occur. Actually I confirm the performance problem is fixed by this
patch.
Takao Indoh (2):
arm64: mm: Restore mm_cpumask (revert commit 38d96287504a ("arm64: mm:
kill mm_cpumask usage"))
arm64: tlb: Add boot parameter to disable TLB flush within the same
inner shareable domain
.../admin-guide/kernel-parameters.txt | 4 +
arch/arm64/include/asm/mmu_context.h | 7 +-
arch/arm64/include/asm/tlbflush.h | 61 ++-----
arch/arm64/kernel/Makefile | 2 +-
arch/arm64/kernel/smp.c | 6 +
arch/arm64/kernel/tlbflush.c | 155 ++++++++++++++++++
arch/arm64/mm/context.c | 2 +
7 files changed, 186 insertions(+), 51 deletions(-)
create mode 100644 arch/arm64/kernel/tlbflush.c
--
2.20.1
