On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: > On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: > > Greetings KVM folks, > > > > I wondering if any information exists for doing SR-IOV on the new VT-d > > capable chipsets with KVM..? From what I understand the patches for > > doing this with KVM are floating around, but I have been unable to find > > any user-level docs for actually making it all go against a upstream > > v2.6.30-rc3 code.. > > > > So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I > > am really hoping to be able to jump to KVM for single-function and and > > then multi-function SR-IOV. I know that the VM migration stuff for IOV > > in Xen is up and running, and I assume it is being worked in for KVM > > instance migration as well..? This part is less important (at least for > > me :-) than getting a stable SR-IOV setup running under the KVM > > hypervisor.. Does anyone have any pointers for this..? > > > > Any comments or suggestions are appreciated! > > > > Hi Nicholas > > The patches are not floating around now. As you know, SR-IOV for Linux have > been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent > released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are > several SRIOV related patches for qemu-kvm, and now they all have been checked > in. > > And for KVM, the extra document is not necessary, for you can simple assign a > VF to guest like any other devices. And how to create VF is specific for each > device driver. So just create a VF then assign it to KVM guest is fine. > Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. >From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Soooo, I enabled the debugging code in kvm-85/qemu/hw/device-assignment.c and see the PAGE aligned MMIO memory for the passed PCIe device is being released during the BUG exceptions above.. Is there something else I should be looking at..? I have pci-stub enabled, and I unbind 02:00.0 from /sys/bus/pci/drivers/e1000e/unbind successfully (just like with Xen and pciback), but I am unable to do the 'echo -n 02:00.0 > /sys/bus/pci/drivers/pci-stub/bind' (it returns write error, no such device, with no dmesg output) on the KVM host running v2.6.30-rc3. Is this supposed to happen on v2.6.30-rc3 with pci-stub..? I am also using the the kvm-85 source dist kvm_intel.ko and kvm.ko kernel modules. Is there something I am missing when building kvm-85 for SR-IOV passthrough..? Also FYI, I am having to use pci=resource_alignment= because the BIOS does not PAGE_SIZE align the MMIO BARs for my multi-function devices.. Also, I tried with disabling the DMAR with the Intel IOMMU passthrough from this patch: https://lists.linux-foundation.org/pipermail/iommu/2009-April/001339.html that did not make it into v2.6.30-rc3. The patch logic was enabled but still I saw the same kvm exceptions from qemu-system-x86_64. Anyways, I am going to give it a shot with the Fedora 11 x86_64 Preview and see if it works as expected with a IOH-5520 chipset with the AMI BIOS on a Tyan S7010 with Xeon 5520s. Hopefully this is just a kvm-85 build and/or install issue I am seeing on my CentOS 5u3 install (that has a Xen PCIe passthrough setup on it as well) with v2.6.30-rc3. I will try on a fresh install on a distro with the new KVM logic and see what happens. :-) Thanks for your comments! --nab -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
