Anyone vaguely familiar with the ARM interrupt architecture (also known as GIC) would certainly agree that it isn't the simplest thing to deal with. Its features are ranging from simple, bare metal interrupt delivery to full blown direct injection into a guest. It is also horribly complex, full of backward[-compatibility] features, and it is very hard to reason about what is going on in the system at any given time. For a hypervisor such as KVM, the GIC is an invasive beast that accounts for a large part of the privileged software we run, as it implements the whole of the architecture with bells and whistles as it tries to cater for all possible guests. At the same time, we have an ongoing effort to make KVM/arm64 a more "verifiable" hypervisor, by allowing only a small amount of code to run at EL2. Moving most of the GIC emulation to userspace would involve sacrificing performance (the architecture really doesn't lend itself to a split model, despite the appearances), and proving that it is actually completely safe is almost an impossible task (I have seen people trying!). Another approach is to bite the bullet, and design from the ground up an interrupt controller that: - works well enough for workloads that mostly deal with virtual interrupts (no device assignment), - is as simple as possible for the hypervisor to implement. Since we cannot retrospectively hack the HW, this is a paravirtualized interrupt controller, where every single operation results in a trap. Yes, it looks like it would be terribly expensive. Or not. The result of the above is a specification from ARM [1] that defines the RVIC and RVID components that make the interrupt controller. It is an *Alpha* spec, to it is very much subject to changes (the hypercall numbers have been redacted out to make that explicit). The result of the above result is this patch series, which provides Linux drivers for rVIC and rVID, as well as a KVM implementation that can be exposed to guests. Most of the patches are a big refactor of the KVM/arm64 code to allow a non-GIC irqchip to be exposed to the guest, as the code that actually deals with delivering interrupts is pretty simple. I intend to carry on refactoring this as more structures could be made irqchip agnostic. We have: - Support for the rVIC/rVID PV interrupt controller architecture in a guest - A large rework of the way the vgic integrates with the rest of KVM, mostly punching a vgic-shaped hole in the code, and replacing it with a set of optional callbacks that an interrupt controller implementation can provide, or not. - A rVIC/rVID implementation for KVM/arm64. This is based on my previously posted IPI-as-IRQ series How does it fare? Well, it's not even bad. There is a bit more overhead than with an actual GIC, but you need to squint really hard to see a difference. Turns out that interacting with a HW interrupt controller isn't free either... Of course, YMMV, and I'd happily look at performance figures if someone has the guts to put them together. What is missing: - Patches for userspace to actually start a rVIC-equipped guest. I have pushed a kvmtool branch at [2]. This is just a terrible pile of hacks, don't trust it to do anything right. It works well enough to spawn a guest with virtio-pci and deliver MSIs though. - DT bindings, which I need to write up - ACPI? Why not... Things that are *not* in the spec: - MSIs. I've made them up in the driver and KVM, and I don't think we can do without them. I intend to feed that requirement back to ARM. - RVID level interrupts. We need them to implement PCI INTx. - Priorities. Not clear whether we really need those, and it would certainly complixify the design. This has been lightly tested on a A55-based system running VHE, and equipped with a GICv2, as well as a couple of nVHE systems with both GICv2 and GICv3. VHE+GICv3 is still untested, as I lack the platform (someone please send me an Ampere Altra box ;-). Patches are based on v5.9-rc3 + the IPI patches, and a branch with everything stacked together is at [3]. [1] https://developer.arm.com/architectures/system-architectures/software-standards/rvic [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/kvmtool.git/log/?h=rvic [3] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/rvic Marc Zyngier (23): irqchip: Add Reduced Virtual Interrupt Controller driver irqchip/rvic: Add support for untrusted interrupt allocation irqchip: Add Reduced Virtual Interrupt Distributor support irqchip/rvid: Add PCI MSI support KVM: arm64: Move GIC model out of the distributor KVM: arm64: vgic-v3: Move early init to kvm_vgic_create() KVM: arm64: Add irqchip callback structure to kvm_arch KVM: arm64: Move kvm_vgic_destroy to kvm_irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_init() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_[un]blocking() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_load/put() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_pending_irq() to irqchip_flow KVM: arm64: Move vgic resource mapping on first run to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_{sync,flush}_hwstate() to irqchip_flow KVM: arm64: nVHE: Only save/restore GICv3 state if modeling a GIC KVM: arm64: Move interrupt injection to irqchip_flow KVM: arm64: Move mapping of HW interrupts into irqchip_flow KVM: arm64: Move set_owner into irqchip_flow KVM: arm64: Turn vgic_initialized into irqchip_finalized KVM: arm64: Move irqfd routing to irqchip_flow KVM: arm64: Tighten msis_require_devid reporting KVM: arm64: Add a rVIC/rVID in-kernel implementation KVM: arm64: Add debugfs files for the rVIC/rVID implementation arch/arm64/include/asm/kvm_host.h | 11 +- arch/arm64/include/asm/kvm_irq.h | 136 +++ arch/arm64/include/uapi/asm/kvm.h | 9 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arch_timer.c | 36 +- arch/arm64/kvm/arm.c | 141 ++- arch/arm64/kvm/hyp/nvhe/switch.c | 12 +- arch/arm64/kvm/hypercalls.c | 7 + arch/arm64/kvm/pmu-emul.c | 10 +- arch/arm64/kvm/rvic-cpu.c | 1213 +++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-debug.c | 7 +- arch/arm64/kvm/vgic/vgic-init.c | 133 ++- arch/arm64/kvm/vgic/vgic-irqfd.c | 72 +- arch/arm64/kvm/vgic/vgic-its.c | 2 +- arch/arm64/kvm/vgic/vgic-kvm-device.c | 18 +- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic-mmio.c | 10 +- arch/arm64/kvm/vgic/vgic-v2.c | 5 - arch/arm64/kvm/vgic/vgic-v3.c | 26 +- arch/arm64/kvm/vgic/vgic.c | 55 +- arch/arm64/kvm/vgic/vgic.h | 37 + drivers/irqchip/Kconfig | 12 + drivers/irqchip/Makefile | 2 + drivers/irqchip/irq-rvic.c | 595 ++++++++++++ drivers/irqchip/irq-rvid.c | 441 +++++++++ include/kvm/arm_rvic.h | 41 + include/kvm/arm_vgic.h | 33 - include/linux/cpuhotplug.h | 1 + include/linux/irqchip/irq-rvic.h | 100 ++ include/uapi/linux/kvm.h | 2 + 30 files changed, 2907 insertions(+), 264 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_irq.h create mode 100644 arch/arm64/kvm/rvic-cpu.c create mode 100644 drivers/irqchip/irq-rvic.c create mode 100644 drivers/irqchip/irq-rvid.c create mode 100644 include/kvm/arm_rvic.h create mode 100644 include/linux/irqchip/irq-rvic.h -- 2.27.0
