This is the second posting of arm64 kernel text replication, rebased on v6.5-rc1. There are a number of issues outstanding from the previous posting: - Making it work with different numbers of page tables / page sizes etc - Using 2-stage page tables to replicate (which needs to be done in EDK2/UEFI rather than the kernel.) The changes since the last version have been: - rox page protection consolidation patch has been merged - Mark's VA-range sanity check patch is now also merged which allows my changes in that area to be dropped. - reordered the first three patches I am not expecting this to be merged, but it would be nice to get a few more eyes and comments on it - and maybe people trying it out. Documentation/admin-guide/kernel-parameters.txt | 5 + arch/arm64/Kconfig | 10 +- arch/arm64/include/asm/cacheflush.h | 2 + arch/arm64/include/asm/ktext.h | 45 ++++++ arch/arm64/include/asm/memory.h | 28 +++- arch/arm64/include/asm/mmu_context.h | 12 +- arch/arm64/include/asm/pgtable.h | 35 ++++- arch/arm64/include/asm/smp.h | 1 + arch/arm64/kernel/alternative.c | 4 +- arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/head.S | 3 +- arch/arm64/kernel/hibernate.c | 2 +- arch/arm64/kernel/kaslr.c | 1 + arch/arm64/kernel/patching.c | 7 +- arch/arm64/kernel/smp.c | 3 + arch/arm64/kernel/suspend.c | 3 +- arch/arm64/kernel/vmlinux.lds.S | 3 + arch/arm64/mm/Makefile | 2 + arch/arm64/mm/init.c | 3 + arch/arm64/mm/ktext.c | 198 ++++++++++++++++++++++++ arch/arm64/mm/mmu.c | 73 +++++++-- 22 files changed, 416 insertions(+), 27 deletions(-) create mode 100644 arch/arm64/include/asm/ktext.h create mode 100644 arch/arm64/mm/ktext.c Original message below. On Tue, May 30, 2023 at 03:04:01PM +0100, Russell King (Oracle) wrote: > Problem > ------- > > NUMA systems have greater latency when accessing data and instructions > across nodes, which can lead to a reduction in performance on CPU cores > that mainly perform accesses beyond their local node. > > Normally when an ARM64 system boots, the kernel will end up placed in > memory, and each CPU core will have to fetch instructions and data from > which ever NUMA node the kernel has been placed. This means that while > executing kernel code, CPUs local to that node will run faster than > CPUs in remote nodes. > > The higher the latency to access remote NUMA node memory, the more the > kernel performance suffers on those nodes. > > If there is a local copy of the kernel text in each node's RAM, and > each node runs the kernel using its local copy of the kernel text, > then it stands to reason that the kernel will run faster due to fewer > stalls while instructions are fetched from remote memory. > > The question then arises how to achieve this. > > Background > ---------- > > An important issue to contend with is what happens when a thread > migrates between nodes. Essentially, the thread's state (including > instruction pointer) is saved to memory, and the scheduler on that CPU > loads some other thread's state and that CPU resumes executing that > new thread. > > The CPU gaining the migrating thread loads the saved state, again > including the instruction pointer, and the gaining CPU resumes fetching > instructions at the virtual address where the original CPU left off. > > The key point is that the virtual address is what matters here, and > this gives us a way to implement kernel text replication fairly easily. > At a practical level, all we need to do is to ensure that the virtual > addresses which contain the kernel text point to a local copy of the > that text. > > This is exactly how this proposal of kernel text replication achieves > the replication. We can go a little bit further and include most of > the read-only data in this replication, as that will never be written > to by the kernel (and thus remains constant.) > > Solution > -------- > > So, what we need to achieve is: > > 1. multiple identical copies of the kernel text (and read-only data) > 2. point the virtual mappings to the appropriate copy of kernel text > for the NUMA node. > > (1) is fairly easy to achieve - we just need to allocate some memory > in the appropriate node and copy the parts of the kernel we want to > replicate. However, we also need to deal with ARM64's kernel patching. > There are two functions that patch the kernel text, > __apply_alternatives() and aarch64_insn_patch_text_nosync(). Both of > these need to to be modified to update all copies of the kernel text. > > (2) is slightly harder. > > Firstly, the aarch64 architecture has a very useful feature here - the > kernel page tables are entirely separate from the user page tables. > The hardware contains two page table pointers, one is used for user > mappings, the other is used for kernel mappings. > > Therefore, we only have one page table to be concerned with: the table > which maps kernel space. We do not need to be concerned with each > user processes page table. > > The approach taken here is to ensure that the kernel is located in an > area of kernel virtual address space covered by a level-0 page table > entry which is not shared with any other user. We can then maintain > separate per-node level-0 page tables for kernel space where the only > difference between them is this level-0 page table entry. > > This gives a couple of benefits. Firstly, when updates to the level-0 > page table happen (e.g. when establishing new mappings) these updates > can simply be copied to the other level-0 page tables provided it isn't > for the kernel image. Secondly, we don't need complexity at lower > levels of the page table code to figure out whether a level-1 or lower > update needs to be propagated to other nodes. > > The level-0 page table entry for the kernel can then be used to point > at a node-unique set of level 1..N page tables to make the appropriate > copy of the kernel text (and read-only data) into kernel space, while > keeping the kernel read-write data shared between nodes. > > Performance Analysis > -------------------- > > Needless to say, the performance results from kernel text replication > are workload specific, but appear to show a gain of between 6% and > 17% for database-centric like workloads. When combined with userspace > awareness of NUMA, this can result in a gain of over 50%. > > Problems > -------- > > There are a few areas that are a problem for kernel text replication: > 1) As this series changes the kernel space virtual address space > layout, it breaks KASAN - and I've zero knowledge of KASAN so I > have no idea how to fix it. I would be grateful for input from > KASAN folk for suggestions how to fix this. > > 2) KASLR can not be used with kernel text replication, since we need > to place the kernel in its own L0 page table entry, not in vmalloc > space. KASLR is disabled when support for kernel text replication > is enabled. > > 3) Changing the kernel virtual address space layout also means that > kaslr_offset() and kaslr_enabled() need to become macros rather > than inline functions due to the use of PGDIR_SIZE in the > calculation of KIMAGE_VADDR. Since asm/pgtable.h defines this > constant, but asm/memory.h is included by asm/pgtable.h, having > this symbol available would produce a circular include > dependency, so I don't think there is any choice here. > > 4) read-only protection for replicated kernel images is not yet > implemented. > > Patch overview: > > Patch 1 cleans up the rox page protection logic. > Patch 2 reoganises the kernel virtual address space layout (causing > problems (1 and 3). > Patch 3 provides a version of cpu_replace_ttbr1 that takes physical > addresses. > Patch 4 makes a needed cache flushing function visible. > Patch 5 through 16 are the guts of kernel text replication. > Patch 17 adds the Kconfig entry for it. > > Further patches not included in this set add a Kconfig for the default > state, a test module, and add code to verify the replicated kernel > text matches the node 0 text after the kernel has completed most of > its boot. > > Documentation/admin-guide/kernel-parameters.txt | 5 + > arch/arm64/Kconfig | 10 +- > arch/arm64/include/asm/cacheflush.h | 2 + > arch/arm64/include/asm/ktext.h | 45 ++++++ > arch/arm64/include/asm/memory.h | 26 ++-- > arch/arm64/include/asm/mmu_context.h | 12 +- > arch/arm64/include/asm/pgtable.h | 35 ++++- > arch/arm64/include/asm/smp.h | 1 + > arch/arm64/kernel/alternative.c | 4 +- > arch/arm64/kernel/asm-offsets.c | 1 + > arch/arm64/kernel/cpufeature.c | 2 +- > arch/arm64/kernel/head.S | 3 +- > arch/arm64/kernel/hibernate.c | 2 +- > arch/arm64/kernel/patching.c | 7 +- > arch/arm64/kernel/smp.c | 3 + > arch/arm64/kernel/suspend.c | 3 +- > arch/arm64/kernel/vmlinux.lds.S | 3 + > arch/arm64/mm/Makefile | 2 + > arch/arm64/mm/init.c | 3 + > arch/arm64/mm/ktext.c | 198 ++++++++++++++++++++++++ > arch/arm64/mm/mmu.c | 85 ++++++++-- > 21 files changed, 413 insertions(+), 39 deletions(-) > create mode 100644 arch/arm64/include/asm/ktext.h > create mode 100644 arch/arm64/mm/ktext.c -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!
