Hi Pavel, On 04/04/2016 12:15 PM, Pavel Fedin wrote: > Hello! > > I'd like to remind that this will break with current vITS code and VFIO. > The problem is short path for raising MSIs via routed GSI. There is a code > which directly calls IRQ injection instead of triggering irqfd. With VFIO > this code is called from within interrupt context. This will cause random > lockups in vits_inject_msi(), which does not use irq-save spin-locking. > I hit this several times and I had to rework ITS emulation in my local > tree. Thank you for the reminder. We will sync with Andre and fix that in the next ITS emulation version. Best Regards Eric > > Kind regards, > Pavel Fedin > Senior Engineer > Samsung Electronics Research center Russia > > > -----Original Message----- > From: Eric Auger [mailto:eric.auger@xxxxxxxxxx] > Sent: Monday, April 04, 2016 11:48 AM > To: eric.auger@xxxxxx; eric.auger@xxxxxxxxxx; christoffer.dall@xxxxxxxxxx; > marc.zyngier@xxxxxxx; andre.przywara@xxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx; > kvm@xxxxxxxxxxxxxxx > Cc: patches@xxxxxxxxxx; pbonzini@xxxxxxxxxx; p.fedin@xxxxxxxxxxx; > Manish.Jaggi@xxxxxxxxxxxxxxxxxx > Subject: [RFC v4 0/7] KVM: arm/arm64: gsi routing support > > With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing > becomes mandated for proper VIRTIO-PCI vhost integration. > > In QEMU, when the VIRTIO-PCI device is programmed with the MSI message, > we previously used direct_mapping trick: this consists in extracting the > SPI ID found in the MSI message and associating an irqfd to that SPI ID. > When vhost worker thread gets a new buffer it signals the irqfd and kvm > then injects this SPI ID on guest. That way although the guest uses MSIs, > no MSI emulation is used. > > This worked fine with GICv2m but does not work anymore with GICV3 ITS. > Indeed this latter implements IRQ translation: what is found in the MSI > message no more is the target SPI ID but is an intermediate event ID used > in the translation process. > > Hence true MSI routing is needed so that the vhost back channel irqfd is > associated to a dummy gsi ID, routed towards the programmed MSI. When KVM > injects the MSI through the in-kernel ITS emulation, the MSI is properly > translated and eventually the SPI ID associated to the event ID is injected > on guest. > > irqchip routing does not sound to be really useful on arm but usage of > MSI routing also mandates to integrate irqchip routing. The initial > implementation of irqfd on arm must be upgraded with the integration > of kvm irqchip.c code and the implementation of its standard hooks > in the architecture specific part. > > In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing > table with flat irqchip routing entries is built enabling to inject gsi > corresponding to the SPI indexes seen by the guest. > > As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this > default routing table and is responsible for building the whole routing > table. > > for arm/arm64 KVM_SET_GSI_ROUTING has a limited support: > - only applies to KVM_IRQFD and not to KVM_IRQ_LINE > > - irqchip routing was tested on Calxeda midway (VFIO with irqfd) > with and without explicit routing > - MSI routing without GICv3 ITS was tested using APM Xgene-I > (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping). > - MSI routing with GICv3 ITS is *NOT* tested. > > Code can be found at > https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.5- > rc6-its-emul-v4-gsi-routing-v4 > > The series applies on top of Andre's branch featuring NEW-VGIC and in-kernel > ITS emulation series: > http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4 > > [1]: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation > http://www.spinics.net/lists/arm-kernel/msg492770.html > [2]: [RFC PATCH 00/45] KVM: arm/arm64: Rework virtual GIC emulation > http://www.spinics.net/lists/arm-kernel/msg492639.html > > GSI flat routing setup on QEMU can be found at: > https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html > > History: > v3 -> v4: > - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not > a stable foundation yet. Hence the revert to RFC. This v4 mostly is a > reflesh/reminder. > - rewrite the cover letter > > v2 -> v3: > - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level > anymore; > use KVM_MSI_VALID_DEVID flag instead > - propagate user flags downto the kernel to make sure the userspace > correctly set devid in GICv3 ITS case (still under discussion) > > v1 -> v2: > - user API changed: > x devid id passed in kvm_irq_routing_msi > x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI > - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct > composed of the msi_msg and devid in kvm_kernel_irq_routing_entry > - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling > in eventfd.c > - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre > - fix bug reported by Andre: bad setting of msi.flags and msi.devid > in kvm_send_userspace_msi > - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq > > RFC -> PATCH: > - clearly state limited support on arm/arm64: > KVM_IRQ_LINE not impacted by GSI routing > - add default routing table feature (new patch file) > - changed uapi to use padding field area > - reword api.txt > > > > Eric Auger (7): > KVM: api: pass the devid in the msi routing entry > KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry > KVM: irqchip: convey devid to kvm_set_msi > KVM: arm/arm64: enable irqchip routing > KVM: arm/arm64: build a default routing table > KVM: arm/arm64: enable MSI routing > KVM: arm: enable KVM_SIGNAL_MSI and MSI routing > > Documentation/virtual/kvm/api.txt | 35 ++++++++++++--- > arch/arm/include/asm/kvm_host.h | 2 + > arch/arm/kvm/Kconfig | 3 ++ > arch/arm/kvm/Makefile | 1 + > arch/arm64/include/asm/kvm_host.h | 1 + > arch/arm64/kvm/Kconfig | 3 ++ > arch/arm64/kvm/Makefile | 1 + > include/kvm/vgic/vgic.h | 2 - > include/linux/kvm_host.h | 8 +++- > include/uapi/linux/kvm.h | 5 ++- > virt/kvm/arm/vgic/vgic-v2.c | 15 +++++++ > virt/kvm/arm/vgic/vgic.c | 7 --- > virt/kvm/arm/vgic/vgic.h | 1 + > virt/kvm/arm/vgic/vgic_init.c | 24 +++++++++++ > virt/kvm/arm/vgic/vgic_irqfd.c | 91 > +++++++++++++++++++++++++++++++-------- > virt/kvm/irqchip.c | 8 +++- > 16 files changed, 170 insertions(+), 37 deletions(-) > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
