Hi, On 22/07/16 17:20, Eric Auger wrote: > With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing > becomes mandated for proper VIRTIO-PCI vhost integration. > > In QEMU, when the VIRTIO-PCI device is programmed with the MSI message, > we previously used direct_mapping trick: this consists in extracting the > IRQ ID found in the MSI message and associating an irqfd to that IRQ ID. > When vhost worker thread gets a new buffer it signals the irqfd and kvm > then injects this IRQ ID on guest. That way although the guest uses MSIs, > no MSI emulation is used. > > This worked fine with GICv2m but does not work anymore with GICV3 ITS. > Indeed this latter implements IRQ translation: what is found in the MSI > message no more is the target IRQ ID but is an intermediate event ID used > in the translation process. > > Hence true MSI routing is needed so that the vhost back channel irqfd is > associated to a dummy gsi ID, routed towards the programmed MSI. When KVM > injects the MSI through the in-kernel ITS emulation, the MSI is properly > translated and eventually the LPI ID associated to the event ID is injected > on guest. > > MSI routing also mandates to integrate irqchip routing. The initial > implementation of irqfd on arm must be upgraded with the integration > of kvm irqchip.c code and the implementation of its standard hooks > in the architecture specific part. > > In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing > table with flat irqchip routing entries is built enabling to inject gsi > corresponding to the SPI indexes seen by the guest. > > As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this > default routing table and is responsible for building the whole routing > table. > > for arm/arm64 KVM_SET_GSI_ROUTING has a limited support: > - only applies to KVM_IRQFD and not to KVM_IRQ_LINE > > - irqchip routing was tested on Calxeda midway (VFIO with irqfd) > with and without explicit routing > - MSI routing was tested on AMD Overdrive and Cavium ThunderX > > Code + dependencies can be found at: > https://github.com/eauger/linux/tree/v4.7-rc7-its-emul-v10-gsi-routing-v9 > > The series depends on > > [1]: [PATCH v10 00/17] KVM: arm64: GICv3 ITS emulation > http://www.spinics.net/lists/kvm/msg135687.html > + [PATCH] KVM: arm/arm64: fix vGICv2 KVM_DEV_ARM_VGIC_GRP_CPU/DIST_REGS > [2]: [PATCH] KVM: arm/arm64: The GIC is dead, long live the GIC > > hence the RFC. > > GSI flat routing setup on QEMU can be found at: > https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html So I gave this series (as part of -next atm) some testing on a Freescale machine. I quickly boot tested the following combinations of options: host kernel: next-20160728 and v4.7.0 guest kernel: next-20160728 and v4.4.0 QEMU: v2.6 and latest HEAD + Eric's ITS & GSI patches (from github) gic-version=2 and gic-version=3 virtio-mmio and virtio-pci kvmtool: latest HEAD and vits-v7 --irqchip=gic-{v2,v3,v3-its} --force-pci and without (virtio-mmio) That summed up to 68 combinations for me, all booting fine. If every part supported ITS and MSIs (host kernel, userland tool, virtio-pci) I saw the ITS emulation being used. I also ran three guests (two QEMU and one kvmtool) in parallel with some interrupt load for some hours. For the records I could easily trigger our infamous kvmtool virtio hang with SMP guests on this machine, but as that happened with older kernels and kvmtools as well, affects only SMP guests and is totally absent when running with QEMU, it's not a regression and unrelated to this series. So FWIW: Tested-by: Andre Przywara <andre.przywara@xxxxxxx> Cheers, Andre > > History: > v8 -> v9: > - remove "[RFC v8 7/7] KVM: arm: Enable KVM_SIGNAL_MSI and MSI routing" > - fix spi id check at IRQ injection > - use default in irqchip switch(ue->type) > > v7 -> v8: > - take into account Radim's comments: > x rework kvm_kernel_irq_routing_entry and remove msi_msg now > replaced by a custom struct > x check user entry flags depending on user entry type > x I kept last patch although challenged by Radim. It can be dropped > according to the consensus. > > v6 -> v7: > - take into account Drew's and Andre's comments > - new patch moving declarations of kvm_setup_default_irq_routing and > kvm_setup_empty_irq_routing outside of kvm_host.h > - vgic_v2m_inject_msi moved in vgic-irqfd > - re-introduce irq.h > - add msi_ prefix to flags/devid kvm_kernel_irq_routing_entry fields > - move kvm_vgic_setup_default_irq_routing declaration in arm_vgic.h > and definition in vgic-irqfd.c > - remove BUG_ON(!vgic_initialized(kvm) in vgic_irqfd_set_irq > - move KVM_IRQCHIP_NUM_PINS in arm_vgic.h and use VGIC_MAX_SPI instead > of 1020 > > v5 -> v6: > - rebase on Andre's v8 + removal of old vgic > - tested on Cavium ThunderX > > V4 -> v5: > - rebase on Andre's v7 + final new vgic code > - check msi->data is within SPI range in vgic_v2m_inject_msi > - squashed enable irq routing and default irqchip table patches > - handle default irqchip table allocation failure > - some rephrasing in doc & comment according to Christoffer's feedbacks > - lock issue reported by Pavel seems to have disappear after 4.2 (MSI injection > fast path) > > v3 -> v4: > - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not > a stable foundation yet. Hence the revert to RFC. This v4 mostly is a > reflesh/reminder. > - rewrite the cover letter > > v2 -> v3: > - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore; > use KVM_MSI_VALID_DEVID flag instead > - propagate user flags downto the kernel to make sure the userspace > correctly set devid in GICv3 ITS case (still under discussion) > > v1 -> v2: > - user API changed: > x devid id passed in kvm_irq_routing_msi > x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI > - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct > composed of the msi_msg and devid in kvm_kernel_irq_routing_entry > - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling > in eventfd.c > - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre > - fix bug reported by Andre: bad setting of msi.flags and msi.devid > in kvm_send_userspace_msi > - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq > > RFC -> PATCH: > - clearly state limited support on arm/arm64: > KVM_IRQ_LINE not impacted by GSI routing > - add default routing table feature (new patch file) > - changed uapi to use padding field area > - reword api.txt > > > Eric Auger (6): > KVM: api: Pass the devid in the msi routing entry > KVM: Add devid in kvm_kernel_irq_routing_entry > KVM: irqchip: Convey devid to kvm_set_msi > KVM: Move kvm_setup_default/empty_irq_routing declaration in arch > specific header > KVM: arm/arm64: Enable irqchip routing > KVM: arm/arm64: Enable MSI routing > > Documentation/virtual/kvm/api.txt | 34 +++++++++--- > arch/arm/kvm/Kconfig | 2 + > arch/arm/kvm/Makefile | 1 + > arch/arm/kvm/irq.h | 19 +++++++ > arch/arm64/kvm/Kconfig | 2 + > arch/arm64/kvm/Makefile | 1 + > arch/arm64/kvm/irq.h | 19 +++++++ > arch/x86/kvm/irq.h | 3 ++ > include/kvm/arm_vgic.h | 7 +++ > include/linux/kvm_host.h | 12 +++-- > include/uapi/linux/kvm.h | 5 +- > virt/kvm/arm/vgic/vgic-init.c | 4 ++ > virt/kvm/arm/vgic/vgic-irqfd.c | 108 +++++++++++++++++++++++++++++++------- > virt/kvm/arm/vgic/vgic.c | 7 --- > virt/kvm/irqchip.c | 28 ++++++---- > 15 files changed, 207 insertions(+), 45 deletions(-) > create mode 100644 arch/arm/kvm/irq.h > create mode 100644 arch/arm64/kvm/irq.h > _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
