Hi Marc, On 5/15/23 19:30, Marc Zyngier wrote: > This is the 4th drop of NV support on arm64 for this year. > > For the previous episodes, see [1]. > > What's changed: > > - New framework to track system register traps that are reinjected in > guest EL2. It is expected to replace the discrete handling we have > enjoyed so far, which didn't scale at all. This has already fixed a > number of bugs that were hidden (a bunch of traps were never > forwarded...). Still a work in progress, but this is going in the > right direction. > > - Allow the L1 hypervisor to have a S2 that has an input larger than > the L0 IPA space. This fixes a number of subtle issues, depending on > how the initial guest was created. > > - Consequently, the patch series has gone longer again. Boo. But > hopefully some of it is easier to review... > > [1] https://lore.kernel.org/r/20230405154008.3552854-1-maz@xxxxxxxxxx I have started testing this and when booting my fedora guest I get [ 151.796544] kvm [7617]: Unsupported guest sys_reg access at: 23f425fd0 [80000209] [ 151.796544] { Op0( 3), Op1( 3), CRn(14), CRm( 3), Op2( 1), func_write }, as soon as the host has kvm-arm.mode=nested This seems to be triggered very early by EDK2 (ArmPkg/Drivers/TimerDxe/TimerDxe.c). If I am not wrong this CNTV_CTL_EL0. Do you have any idea? By the way I got this already with your v9 (git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/nv-6.4-WIP) and with your v10 (I cherry-picked your maz/kvm-arm64/nv-6.5-WIP branch). Thanks Eric > > Andre Przywara (1): > KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ > > Christoffer Dall (5): > KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 > KVM: arm64: nv: Implement nested Stage-2 page table walk logic > KVM: arm64: nv: Unmap/flush shadow stage 2 page tables > KVM: arm64: nv: vgic: Emulate the HW bit in software > KVM: arm64: nv: Sync nested timer state with FEAT_NV2 > > Jintack Lim (7): > KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2 > KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting > KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings > KVM: arm64: nv: Respect virtual HCR_EL2.{NV,TSC) settings > KVM: arm64: nv: Configure HCR_EL2 for nested virtualization > KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2 > KVM: arm64: nv: Nested GICv3 Support > > Marc Zyngier (46): > KVM: arm64: Move VTCR_EL2 into struct s2_mmu > arm64: Add missing Set/Way CMO encodings > arm64: Add missing VA CMO encodings > arm64: Add missing ERXMISCx_EL1 encodings > arm64: Add missing DC ZVA/GVA/GZVA encodings > arm64: Add TLBI operation encodings > arm64: Add AT operation encodings > KVM: arm64: Add missing HCR_EL2 trap bits > KVM: arm64: nv: Add trap forwarding infrastructure > KVM: arm64: nv: Add trap forwarding for HCR_EL2 > KVM: arm64: nv: Expose FEAT_EVT to nested guests > KVM: arm64: nv: Add trap forwarding for MDCR_EL2 > KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2 > KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers > KVM: arm64: nv: Handle virtual EL2 registers in > vcpu_read/write_sys_reg() > KVM: arm64: nv: Handle SPSR_EL2 specially > KVM: arm64: nv: Handle HCR_EL2.E2H specially > KVM: arm64: nv: Save/Restore vEL2 sysregs > KVM: arm64: nv: Support multiple nested Stage-2 mmu structures > KVM: arm64: nv: Handle shadow stage 2 page faults > KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's > KVM: arm64: nv: Set a handler for the system instruction traps > KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 > KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's > KVM: arm64: nv: Hide RAS from nested guests > KVM: arm64: nv: Add handling of EL2-specific timer registers > KVM: arm64: nv: Load timer before the GIC > KVM: arm64: nv: Don't load the GICv4 context on entering a nested > guest > KVM: arm64: nv: Implement maintenance interrupt forwarding > KVM: arm64: nv: Deal with broken VGIC on maintenance interrupt > delivery > KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT > KVM: arm64: nv: Add handling of FEAT_TTL TLB invalidation > KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like > information > KVM: arm64: nv: Tag shadow S2 entries with nested level > KVM: arm64: nv: Add include containing the VNCR_EL2 offsets > KVM: arm64: nv: Map VNCR-capable registers to a separate page > KVM: arm64: nv: Move nested vgic state into the sysreg file > KVM: arm64: Add FEAT_NV2 cpu feature > KVM: arm64: nv: Fold GICv3 host trapping requirements into guest setup > KVM: arm64: nv: Publish emulated timer interrupt state in the > in-memory state > KVM: arm64: nv: Allocate VNCR page when required > KVM: arm64: nv: Enable ARMv8.4-NV support > KVM: arm64: nv: Fast-track 'InHost' exception returns > KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests > KVM: arm64: nv: Use FEAT_ECV to trap access to EL0 timers > KVM: arm64: nv: Accelerate EL0 timer read accesses when FEAT_ECV is on > > .../virt/kvm/devices/arm-vgic-v3.rst | 12 +- > arch/arm64/include/asm/esr.h | 1 + > arch/arm64/include/asm/kvm_arm.h | 14 + > arch/arm64/include/asm/kvm_asm.h | 4 + > arch/arm64/include/asm/kvm_emulate.h | 93 +- > arch/arm64/include/asm/kvm_host.h | 181 +++- > arch/arm64/include/asm/kvm_hyp.h | 2 + > arch/arm64/include/asm/kvm_mmu.h | 20 +- > arch/arm64/include/asm/kvm_nested.h | 133 +++ > arch/arm64/include/asm/stage2_pgtable.h | 4 +- > arch/arm64/include/asm/sysreg.h | 196 ++++ > arch/arm64/include/asm/vncr_mapping.h | 74 ++ > arch/arm64/include/uapi/asm/kvm.h | 1 + > arch/arm64/kernel/cpufeature.c | 11 + > arch/arm64/kvm/Makefile | 4 +- > arch/arm64/kvm/arch_timer.c | 98 +- > arch/arm64/kvm/arm.c | 33 +- > arch/arm64/kvm/at.c | 219 ++++ > arch/arm64/kvm/emulate-nested.c | 934 ++++++++++++++++- > arch/arm64/kvm/handle_exit.c | 29 +- > arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +- > arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 5 +- > arch/arm64/kvm/hyp/nvhe/mem_protect.c | 8 +- > arch/arm64/kvm/hyp/nvhe/pkvm.c | 4 +- > arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- > arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- > arch/arm64/kvm/hyp/pgtable.c | 2 +- > arch/arm64/kvm/hyp/vgic-v3-sr.c | 6 +- > arch/arm64/kvm/hyp/vhe/switch.c | 206 +++- > arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 124 ++- > arch/arm64/kvm/hyp/vhe/tlb.c | 83 ++ > arch/arm64/kvm/mmu.c | 255 ++++- > arch/arm64/kvm/nested.c | 799 ++++++++++++++- > arch/arm64/kvm/pkvm.c | 2 +- > arch/arm64/kvm/reset.c | 7 + > arch/arm64/kvm/sys_regs.c | 958 +++++++++++++++++- > arch/arm64/kvm/trace_arm.h | 19 + > arch/arm64/kvm/vgic/vgic-init.c | 33 + > arch/arm64/kvm/vgic/vgic-kvm-device.c | 32 +- > arch/arm64/kvm/vgic/vgic-v3-nested.c | 248 +++++ > arch/arm64/kvm/vgic/vgic-v3.c | 43 +- > arch/arm64/kvm/vgic/vgic.c | 29 + > arch/arm64/kvm/vgic/vgic.h | 10 + > arch/arm64/tools/cpucaps | 1 + > include/clocksource/arm_arch_timer.h | 4 + > include/kvm/arm_arch_timer.h | 1 + > include/kvm/arm_vgic.h | 17 + > include/uapi/linux/kvm.h | 1 + > tools/arch/arm/include/uapi/asm/kvm.h | 1 + > 49 files changed, 4790 insertions(+), 183 deletions(-) > create mode 100644 arch/arm64/include/asm/vncr_mapping.h > create mode 100644 arch/arm64/kvm/at.c > create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c >
