This is my first upstream feature submission so please go easy ;-) Add support for FEAT_LPA2 to KVM for both hypervisor stage 1 (for the nvhe/protected modes) and the vm stage 2 translation tables. FEAT_LPA2 enables 52 bit PAs and VAs for 4KB and 16KB granules (note this is already supported for 64KB granules via the FEAT_LPA and FEAT_LVA extensions). The series does not include support for FEAT_LPA2 in the kernel stage 1. This support is provided separately by Ard Biesheuvel's series at [1]. Although my series does not rely on Ard's work and I'm posting the patches based on top of v6.1-rc6, I have tested with and without Ard's changes and provide the results below. The testing highlighted some pre-existing bugs in mainline and I have submitted fixes for these separately at [2], [3], [4]. You can find the tree without Ard's changes at [5] and a tree with Ard's changes at [6]. The series is broken up into 3 phases; update TLBI routines, support 52-bit output addresses, and support 52 bit input addresses. Update TLBI Routines: The update to the range-based TLBI instructions is technically not needed for the KVM support because KVM only uses the non-range-based TBLI instructions as far as I can see. But I've done both parts because I thought it was sensible to do all the updates together - the range-based stuff will be needed by Ard's patch I think. See commit message for details. Support 52-bit Output Addresses: FEAT_LPA2 changes the format of the PTEs. The HW advertises support for LPA2 independently for stage 1 and stage 2, and therefore its possible to have it for one and not the other. I've assumed that there is a valid case for this if stage 1 is not supported but stage 2 is, KVM could still then use LPA2 at stage 2 to create a 52 bit IPA space (which could then be consumed by a 64KB page guest kernel with the help of FEAT_LPA). Because of this independence and the fact that the kvm pgtable library is used for both stage 1 and stage 2 tables, this means the library now has to remember the in-use format on a per-page-table basis. To do this, I had to rework some functions to take a `struct kvm_pgtable *` parameter, and as a result, there is a noisy patch to add this parameter. Support 52-bit Input Addresses: The main difficulty here is that at stage 1 for 4KB pages, 52-bit IA requires a extra level of lookup, and that level is called '-1'. (Although stage 2 can use concatenated pages at the first level, and therefore still only uses 4 levels, the kvm pgtable library deals with both stage 1 and stage 2 tables). So there is another noisy patch to convert all level variables to signed. This is all tested on the FVP, using a test harness I put together, which does a host + guest boot test for 180 configurations, built from all the (valid) combinations of various FVP, host kernel and guest kernel parameters: - hw_pa: [48, lpa, lpa2] - hw_va: [48, 52] - kvm_mode: [vhe, nvhe, protected] - host_page_size: [4KB, 16KB, 64KB] - host_pa: [48, 52] - host_va: [48, 52] - host_load_addr: [low, high] - guest_page_size: [64KB] - guest_pa: [52] - guest_va: [52] - guest_load_addr: [low, high] I provide the results for the tree at [5] (which doesn't contain Ard's series), where all tests pass except the 12 cases where a 4KB or 16KB host kernel is attempting to boot in high memory - these are expected to fail due to the host kernel not supporting LPA2. When running the tests against the tree at [6] (which does contain Ard's series plus a few minor fixes), all tests pass. (I'm omitting guest_page_size, guest_va and guest_pa below since it is always 64KB/52/52). +-------+-------+-----------+----------------+---------+---------+----------------+-----------------+-------+ | hw_pa | hw_va | kvm_mode | host_page_size | host_va | host_pa | host_load_addr | guest_load_addr | pass | +-------+-------+-----------+----------------+---------+---------+----------------+-----------------+-------+ | 48 | 48 | vhe | 4k | 48 | 48 | low | low | True | | 48 | 48 | vhe | 4k | 52 | 52 | low | low | True | | 48 | 48 | vhe | 16k | 48 | 48 | low | low | True | | 48 | 48 | vhe | 16k | 52 | 52 | low | low | True | | 48 | 48 | vhe | 64k | 48 | 48 | low | low | True | | 48 | 48 | vhe | 64k | 48 | 52 | low | low | True | | 48 | 48 | vhe | 64k | 52 | 48 | low | low | True | | 48 | 48 | vhe | 64k | 52 | 52 | low | low | True | | 48 | 48 | nvhe | 4k | 48 | 48 | low | low | True | | 48 | 48 | nvhe | 4k | 52 | 52 | low | low | True | | 48 | 48 | nvhe | 16k | 48 | 48 | low | low | True | | 48 | 48 | nvhe | 16k | 52 | 52 | low | low | True | | 48 | 48 | nvhe | 64k | 48 | 48 | low | low | True | | 48 | 48 | nvhe | 64k | 48 | 52 | low | low | True | | 48 | 48 | nvhe | 64k | 52 | 48 | low | low | True | | 48 | 48 | nvhe | 64k | 52 | 52 | low | low | True | | 48 | 48 | protected | 4k | 48 | 48 | low | low | True | | 48 | 48 | protected | 4k | 52 | 52 | low | low | True | | 48 | 48 | protected | 16k | 48 | 48 | low | low | True | | 48 | 48 | protected | 16k | 52 | 52 | low | low | True | | 48 | 48 | protected | 64k | 48 | 48 | low | low | True | | 48 | 48 | protected | 64k | 48 | 52 | low | low | True | | 48 | 48 | protected | 64k | 52 | 48 | low | low | True | | 48 | 48 | protected | 64k | 52 | 52 | low | low | True | | 48 | 52 | vhe | 4k | 48 | 48 | low | low | True | | 48 | 52 | vhe | 4k | 52 | 52 | low | low | True | | 48 | 52 | vhe | 16k | 48 | 48 | low | low | True | | 48 | 52 | vhe | 16k | 52 | 52 | low | low | True | | 48 | 52 | vhe | 64k | 48 | 48 | low | low | True | | 48 | 52 | vhe | 64k | 48 | 52 | low | low | True | | 48 | 52 | vhe | 64k | 52 | 48 | low | low | True | | 48 | 52 | vhe | 64k | 52 | 52 | low | low | True | | 48 | 52 | nvhe | 4k | 48 | 48 | low | low | True | | 48 | 52 | nvhe | 4k | 52 | 52 | low | low | True | | 48 | 52 | nvhe | 16k | 48 | 48 | low | low | True | | 48 | 52 | nvhe | 16k | 52 | 52 | low | low | True | | 48 | 52 | nvhe | 64k | 48 | 48 | low | low | True | | 48 | 52 | nvhe | 64k | 48 | 52 | low | low | True | | 48 | 52 | nvhe | 64k | 52 | 48 | low | low | True | | 48 | 52 | nvhe | 64k | 52 | 52 | low | low | True | | 48 | 52 | protected | 4k | 48 | 48 | low | low | True | | 48 | 52 | protected | 4k | 52 | 52 | low | low | True | | 48 | 52 | protected | 16k | 48 | 48 | low | low | True | | 48 | 52 | protected | 16k | 52 | 52 | low | low | True | | 48 | 52 | protected | 64k | 48 | 48 | low | low | True | | 48 | 52 | protected | 64k | 48 | 52 | low | low | True | | 48 | 52 | protected | 64k | 52 | 48 | low | low | True | | 48 | 52 | protected | 64k | 52 | 52 | low | low | True | | lpa | 48 | vhe | 4k | 48 | 48 | low | low | True | | lpa | 48 | vhe | 4k | 52 | 52 | low | low | True | | lpa | 48 | vhe | 16k | 48 | 48 | low | low | True | | lpa | 48 | vhe | 16k | 52 | 52 | low | low | True | | lpa | 48 | vhe | 64k | 48 | 48 | low | low | True | | lpa | 48 | vhe | 64k | 48 | 52 | low | low | True | | lpa | 48 | vhe | 64k | 52 | 48 | low | low | True | | lpa | 48 | vhe | 64k | 52 | 52 | low | low | True | | lpa | 48 | nvhe | 4k | 48 | 48 | low | low | True | | lpa | 48 | nvhe | 4k | 52 | 52 | low | low | True | | lpa | 48 | nvhe | 16k | 48 | 48 | low | low | True | | lpa | 48 | nvhe | 16k | 52 | 52 | low | low | True | | lpa | 48 | nvhe | 64k | 48 | 48 | low | low | True | | lpa | 48 | nvhe | 64k | 48 | 52 | low | low | True | | lpa | 48 | nvhe | 64k | 52 | 48 | low | low | True | | lpa | 48 | nvhe | 64k | 52 | 52 | low | low | True | | lpa | 48 | protected | 4k | 48 | 48 | low | low | True | | lpa | 48 | protected | 4k | 52 | 52 | low | low | True | | lpa | 48 | protected | 16k | 48 | 48 | low | low | True | | lpa | 48 | protected | 16k | 52 | 52 | low | low | True | | lpa | 48 | protected | 64k | 48 | 48 | low | low | True | | lpa | 48 | protected | 64k | 48 | 52 | low | low | True | | lpa | 48 | protected | 64k | 52 | 48 | low | low | True | | lpa | 48 | protected | 64k | 52 | 52 | low | low | True | | lpa | 52 | vhe | 4k | 48 | 48 | low | low | True | | lpa | 52 | vhe | 4k | 52 | 52 | low | low | True | | lpa | 52 | vhe | 16k | 48 | 48 | low | low | True | | lpa | 52 | vhe | 16k | 52 | 52 | low | low | True | | lpa | 52 | vhe | 64k | 48 | 48 | low | low | True | | lpa | 52 | vhe | 64k | 48 | 48 | low | high | True | | lpa | 52 | vhe | 64k | 48 | 52 | low | low | True | | lpa | 52 | vhe | 64k | 48 | 52 | low | high | True | | lpa | 52 | vhe | 64k | 52 | 48 | low | low | True | | lpa | 52 | vhe | 64k | 52 | 48 | low | high | True | | lpa | 52 | vhe | 64k | 52 | 52 | low | low | True | | lpa | 52 | vhe | 64k | 52 | 52 | low | high | True | | lpa | 52 | vhe | 64k | 52 | 52 | high | low | True | | lpa | 52 | vhe | 64k | 52 | 52 | high | high | True | | lpa | 52 | nvhe | 4k | 48 | 48 | low | low | True | | lpa | 52 | nvhe | 4k | 52 | 52 | low | low | True | | lpa | 52 | nvhe | 16k | 48 | 48 | low | low | True | | lpa | 52 | nvhe | 16k | 52 | 52 | low | low | True | | lpa | 52 | nvhe | 64k | 48 | 48 | low | low | True | | lpa | 52 | nvhe | 64k | 48 | 48 | low | high | True | | lpa | 52 | nvhe | 64k | 48 | 52 | low | low | True | | lpa | 52 | nvhe | 64k | 48 | 52 | low | high | True | | lpa | 52 | nvhe | 64k | 52 | 48 | low | low | True | | lpa | 52 | nvhe | 64k | 52 | 48 | low | high | True | | lpa | 52 | nvhe | 64k | 52 | 52 | low | low | True | | lpa | 52 | nvhe | 64k | 52 | 52 | low | high | True | | lpa | 52 | nvhe | 64k | 52 | 52 | high | low | True | | lpa | 52 | nvhe | 64k | 52 | 52 | high | high | True | | lpa | 52 | protected | 4k | 48 | 48 | low | low | True | | lpa | 52 | protected | 4k | 52 | 52 | low | low | True | | lpa | 52 | protected | 16k | 48 | 48 | low | low | True | | lpa | 52 | protected | 16k | 52 | 52 | low | low | True | | lpa | 52 | protected | 64k | 48 | 48 | low | low | True | | lpa | 52 | protected | 64k | 48 | 48 | low | high | True | | lpa | 52 | protected | 64k | 48 | 52 | low | low | True | | lpa | 52 | protected | 64k | 48 | 52 | low | high | True | | lpa | 52 | protected | 64k | 52 | 48 | low | low | True | | lpa | 52 | protected | 64k | 52 | 48 | low | high | True | | lpa | 52 | protected | 64k | 52 | 52 | low | low | True | | lpa | 52 | protected | 64k | 52 | 52 | low | high | True | | lpa | 52 | protected | 64k | 52 | 52 | high | low | True | | lpa | 52 | protected | 64k | 52 | 52 | high | high | True | | lpa2 | 52 | vhe | 4k | 48 | 48 | low | low | True | | lpa2 | 52 | vhe | 4k | 48 | 48 | low | high | True | | lpa2 | 52 | vhe | 4k | 52 | 52 | low | low | True | | lpa2 | 52 | vhe | 4k | 52 | 52 | low | high | True | | lpa2 | 52 | vhe | 4k | 52 | 52 | high | low | False | | lpa2 | 52 | vhe | 4k | 52 | 52 | high | high | False | | lpa2 | 52 | vhe | 16k | 48 | 48 | low | low | True | | lpa2 | 52 | vhe | 16k | 48 | 48 | low | high | True | | lpa2 | 52 | vhe | 16k | 52 | 52 | low | low | True | | lpa2 | 52 | vhe | 16k | 52 | 52 | low | high | True | | lpa2 | 52 | vhe | 16k | 52 | 52 | high | low | False | | lpa2 | 52 | vhe | 16k | 52 | 52 | high | high | False | | lpa2 | 52 | vhe | 64k | 48 | 48 | low | low | True | | lpa2 | 52 | vhe | 64k | 48 | 48 | low | high | True | | lpa2 | 52 | vhe | 64k | 48 | 52 | low | low | True | | lpa2 | 52 | vhe | 64k | 48 | 52 | low | high | True | | lpa2 | 52 | vhe | 64k | 52 | 48 | low | low | True | | lpa2 | 52 | vhe | 64k | 52 | 48 | low | high | True | | lpa2 | 52 | vhe | 64k | 52 | 52 | low | low | True | | lpa2 | 52 | vhe | 64k | 52 | 52 | low | high | True | | lpa2 | 52 | vhe | 64k | 52 | 52 | high | low | True | | lpa2 | 52 | vhe | 64k | 52 | 52 | high | high | True | | lpa2 | 52 | nvhe | 4k | 48 | 48 | low | low | True | | lpa2 | 52 | nvhe | 4k | 48 | 48 | low | high | True | | lpa2 | 52 | nvhe | 4k | 52 | 52 | low | low | True | | lpa2 | 52 | nvhe | 4k | 52 | 52 | low | high | True | | lpa2 | 52 | nvhe | 4k | 52 | 52 | high | low | False | | lpa2 | 52 | nvhe | 4k | 52 | 52 | high | high | False | | lpa2 | 52 | nvhe | 16k | 48 | 48 | low | low | True | | lpa2 | 52 | nvhe | 16k | 48 | 48 | low | high | True | | lpa2 | 52 | nvhe | 16k | 52 | 52 | low | low | True | | lpa2 | 52 | nvhe | 16k | 52 | 52 | low | high | True | | lpa2 | 52 | nvhe | 16k | 52 | 52 | high | low | False | | lpa2 | 52 | nvhe | 16k | 52 | 52 | high | high | False | | lpa2 | 52 | nvhe | 64k | 48 | 48 | low | low | True | | lpa2 | 52 | nvhe | 64k | 48 | 48 | low | high | True | | lpa2 | 52 | nvhe | 64k | 48 | 52 | low | low | True | | lpa2 | 52 | nvhe | 64k | 48 | 52 | low | high | True | | lpa2 | 52 | nvhe | 64k | 52 | 48 | low | low | True | | lpa2 | 52 | nvhe | 64k | 52 | 48 | low | high | True | | lpa2 | 52 | nvhe | 64k | 52 | 52 | low | low | True | | lpa2 | 52 | nvhe | 64k | 52 | 52 | low | high | True | | lpa2 | 52 | nvhe | 64k | 52 | 52 | high | low | True | | lpa2 | 52 | nvhe | 64k | 52 | 52 | high | high | True | | lpa2 | 52 | protected | 4k | 48 | 48 | low | low | True | | lpa2 | 52 | protected | 4k | 48 | 48 | low | high | True | | lpa2 | 52 | protected | 4k | 52 | 52 | low | low | True | | lpa2 | 52 | protected | 4k | 52 | 52 | low | high | True | | lpa2 | 52 | protected | 4k | 52 | 52 | high | low | False | | lpa2 | 52 | protected | 4k | 52 | 52 | high | high | False | | lpa2 | 52 | protected | 16k | 48 | 48 | low | low | True | | lpa2 | 52 | protected | 16k | 48 | 48 | low | high | True | | lpa2 | 52 | protected | 16k | 52 | 52 | low | low | True | | lpa2 | 52 | protected | 16k | 52 | 52 | low | high | True | | lpa2 | 52 | protected | 16k | 52 | 52 | high | low | False | | lpa2 | 52 | protected | 16k | 52 | 52 | high | high | False | | lpa2 | 52 | protected | 64k | 48 | 48 | low | low | True | | lpa2 | 52 | protected | 64k | 48 | 48 | low | high | True | | lpa2 | 52 | protected | 64k | 48 | 52 | low | low | True | | lpa2 | 52 | protected | 64k | 48 | 52 | low | high | True | | lpa2 | 52 | protected | 64k | 52 | 48 | low | low | True | | lpa2 | 52 | protected | 64k | 52 | 48 | low | high | True | | lpa2 | 52 | protected | 64k | 52 | 52 | low | low | True | | lpa2 | 52 | protected | 64k | 52 | 52 | low | high | True | | lpa2 | 52 | protected | 64k | 52 | 52 | high | low | True | | lpa2 | 52 | protected | 64k | 52 | 52 | high | high | True | +-------+-------+-----------+----------------+---------+---------+----------------+-----------------+-------+ [1] https://lore.kernel.org/linux-arm-kernel/20221124123932.2648991-1-ardb@xxxxxxxxxx [2] https://lore.kernel.org/kvmarm/20221027120945.29679-1-ryan.roberts@xxxxxxx [3] https://lore.kernel.org/kvmarm/20221103150507.32948-1-ryan.roberts@xxxxxxx [4] https://lore.kernel.org/kvmarm/20221205114031.3972780-1-ryan.roberts@xxxxxxx [5] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/lpa2/kvm_lkml-v1 [6] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/lpa2/ardb_arm64-4k-lpa2_plus_kvm_2022-12-01 Thanks, Ryan Anshuman Khandual (1): arm64/mm: Add FEAT_LPA2 specific ID_AA64MMFR0.TGRAN[2] Ryan Roberts (11): arm64/mm: Update tlb invalidation routines for FEAT_LPA2 KVM: arm64: Add new (V)TCR_EL2 field definitions for FEAT_LPA2 KVM: arm64: Plumbing to enable multiple pgtable formats KVM: arm64: Maintain page-table format info in struct kvm_pgtable KVM: arm64: Use LPA2 page-tables for stage2 if HW supports it KVM: arm64: Use LPA2 page-tables for hyp stage1 if HW supports it KVM: arm64: Insert PS field at TCR_EL2 assembly time KVM: arm64: Convert translation level parameter to s8 KVM: arm64: Rework logic to en/decode VTCR_EL2.{SL0, SL2} fields KVM: arm64: Support upto 5 levels of translation in kvm_pgtable KVM: arm64: Allow guests with >48-bit IPA size on FEAT_LPA2 systems arch/arm64/include/asm/kvm_arm.h | 79 +++--- arch/arm64/include/asm/kvm_emulate.h | 14 +- arch/arm64/include/asm/kvm_pgtable.h | 131 +++++++-- arch/arm64/include/asm/kvm_pkvm.h | 5 +- arch/arm64/include/asm/pgtable-prot.h | 6 + arch/arm64/include/asm/stage2_pgtable.h | 13 +- arch/arm64/include/asm/sysreg.h | 5 + arch/arm64/include/asm/tlb.h | 15 +- arch/arm64/include/asm/tlbflush.h | 83 ++++-- arch/arm64/kvm/arm.c | 5 + arch/arm64/kvm/hyp/nvhe/hyp-init.S | 4 - arch/arm64/kvm/hyp/nvhe/mem_protect.c | 21 +- arch/arm64/kvm/hyp/nvhe/setup.c | 28 +- arch/arm64/kvm/hyp/pgtable.c | 354 +++++++++++++++--------- arch/arm64/kvm/mmu.c | 15 +- arch/arm64/kvm/reset.c | 11 +- 16 files changed, 525 insertions(+), 264 deletions(-) -- 2.25.1 _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
