Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu On Mon, May 04, 2009 at 06:40:36PM +0800, Nicholas A. Bellinger wrote: > On Mon, 2009-05-04 at 17:49 +0800, Sheng Yang wrote: > > On Monday 04 May 2009 17:11:59 Nicholas A. Bellinger wrote: > > > On Mon, 2009-05-04 at 16:20 +0800, Sheng Yang wrote: > > > > On Monday 04 May 2009 12:36:04 Nicholas A. Bellinger wrote: > > > > > On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: > > > > > > On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: > > > > > > > On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: > > > > > > > > On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger > > wrote: > > > > > > > > > Greetings KVM folks, > > > > > > > > > > > > > > > > > > I wondering if any information exists for doing SR-IOV on the > > > > > > > > > new VT-d capable chipsets with KVM..? From what I understand > > > > > > > > > the patches for doing this with KVM are floating around, but I > > > > > > > > > have been unable to find any user-level docs for actually > > > > > > > > > making it all go against a upstream v2.6.30-rc3 code.. > > > > > > > > > > > > > > > > > > So far I have been doing IOV testing with Xen 3.3 and > > > > > > > > > 3.4.0-pre, and I am really hoping to be able to jump to KVM for > > > > > > > > > single-function and and then multi-function SR-IOV. I know > > > > > > > > > that the VM migration stuff for IOV in Xen is up and running, > > > > > > > > > and I assume it is being worked in for KVM instance migration > > > > > > > > > as well..? This part is less important (at least for me :-) > > > > > > > > > than getting a stable SR-IOV setup running under the KVM > > > > > > > > > hypervisor.. Does anyone have any pointers for this..? > > > > > > > > > > > > > > > > > > Any comments or suggestions are appreciated! > > > > > > > > > > > > > > > > Hi Nicholas > > > > > > > > > > > > > > > > The patches are not floating around now. As you know, SR-IOV for > > > > > > > > Linux have been in 2.6.30, so then you can use upstream KVM and > > > > > > > > qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host > > > > > > > > kernel. And some time ago, there are several SRIOV related > > > > > > > > patches for qemu-kvm, and now they all have been checked in. > > > > > > > > > > > > > > > > And for KVM, the extra document is not necessary, for you can > > > > > > > > simple assign a VF to guest like any other devices. And how to > > > > > > > > create VF is specific for each device driver. So just create a VF > > > > > > > > then assign it to KVM guest is fine. > > > > > > > > > > > > > > Greetings Sheng, > > > > > > > > > > > > > > So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 > > > > > > > checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel > > > > > > > IOH-5520 based dual socket Nehalem board. I have enabled DMAR and > > > > > > > Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I > > > > > > > can tell, the KVM_CAP_* defines from libkvm are enabled with > > > > > > > building kvm-85 after './configure > > > > > > > --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is > > > > > > > being enabled in > > > > > > > kvm-85/qemu/hw/device-assignment.c AFAICT.. > > > > > > > > > > > > > > >From there, I use the freshly installed qemu-x86_64-system binary > > > > > > > > to > > > > > > > > > > > > > > start a Debian 5 x86_64 HVM (that previously had been moving > > > > > > > network packets under Xen for PCIe passthrough). I see the MSI-X > > > > > > > interrupt remapping working on the KVM host for the passed > > > > > > > -pcidevice, and the MMIO mappings from the qemu build that I also > > > > > > > saw while using Xen/qemu-dm built with PCI passthrough are there as > > > > > > > well.. > > > > > > > > > > > > Hi Nicholas > > > > > > > > > > > > > But while the KVM guest is booting, I see the following > > > > > > > exception(s) from qemu-x86_64-system for one of the VFs for a > > > > > > > multi-function PCIe device: > > > > > > > > > > > > > > BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) > > > > > > > > > > > > This one is mostly harmless. > > > > > > > > > > Ok, good to know.. :-) > > > > > > > > > > > > I try with one of the on-board e1000e ports (02:00.0) and I see the > > > > > > > same exception along with some MSI-X exceptions from > > > > > > > qemu-x86_64-system in KVM guest.. However, I am still able to see > > > > > > > the e1000e and the other vxge multi-function device with lspci, but > > > > > > > I am unable to dhcp or ping with the e1000e and VF from > > > > > > > multi-function device fails to register the MSI-X interrupt in the > > > > > > > guest.. > > > > > > > > > > > > Did you see the interrupt in the guest and host side? > > > > > > > > > > Ok, I am restarting the e1000e test with a fresh Fedora 11 install and > > > > > KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and > > > > > attaching the e1000e single-function device at 02:00.0 to pci-stub > > > > > with: > > > > > > > > > > echo "8086 10d3" > /sys/bus/pci/drivers/pci-stub/new_id > > > > > echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind > > > > > echo 0000:02:00.0 > /sys/bus/pci/drivers/pci-stub/bind > > > > > > > > > > I see the following the KVM host kernel ring buffer: > > > > > > > > > > e1000e 0000:02:00.0: PCI INT A disabled > > > > > pci-stub 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17 > > > > > pci-stub 0000:02:00.0: irq 58 for MSI/MSI-X > > > > > > > > > > > I think you can try on- > > > > > > board e1000e for MSI-X first. And please ensure correlated driver > > > > > > have been loaded correctly. > > > > > > > > > > <nod>.. > > > > > > > > > > > And what do you mean by "some MSI-X exceptions"? Better with > > > > > > the log. > > > > > > > > > > Ok, with the Fedora 11 installed qemu-kemu, I see the expected > > > > > kvm_destroy_phys_mem() statements: > > > > > > > > > > #kvm-host qemu-kvm -m 2048 -smp 8 -pcidevice host=02:00.0 > > > > > lenny64guest1-orig.img BUG: kvm_destroy_phys_mem: invalid parameters > > > > > (slot=-1) > > > > > BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) > > > > > > > > > > However I still see the following in the KVM guest kernel ring buffer > > > > > running v2.6.30-rc in the HVM guest. > > > > > > > > > > [ 5.523790] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 > > > > > [ 5.524582] e1000e 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 > > > > > (level, high) -> IRQ 10 [ 5.525710] e1000e 0000:00:05.0: setting > > > > > latency timer to 64 > > > > > [ 5.526048] 0000:00:05.0: 0000:00:05.0: Failed to initialize MSI-X > > > > > interrupts. Falling back to MSI interrupts. [ 5.527200] > > > > > 0000:00:05.0: 0000:00:05.0: Failed to initialize MSI interrupts. > > > > > Falling back to legacy interrupts. [ 5.829988] 0000:00:05.0: eth0: > > > > > (PCI Express:2.5GB/s:Width x1) 00:e0:81:c0:90:b2 [ 5.830672] > > > > > 0000:00:05.0: eth0: Intel(R) PRO/1000 Network Connection [ 5.831240] > > > > > 0000:00:05.0: eth0: MAC: 3, PHY: 8, PBA No: ffffff-0ff > > > > > > > > Hi Nicholas > > > > > > > > I think something need to be clarify: > > > > 1. For SRIOV, you need 2.6.30 as host kernel... But it's better to know > > > > if normal device assignment work in your environment at first. > > > > 2. The Fedora's userspace is even more old... You'd better try qemu-kvm > > > > upstream, which is more convenient for us to track the problem(and kvm-85 > > > > is also ok). And as you see above, your QEmu don't support MSI/MSIX... > > > > > > Ok, got it.. > > > > > > > So you can: > > > > 1. Use latest qemu-kvm or kvm-85's QEmu. As well as latest KVM. > > > > > > Ok, I am now updated on in the FC 11 Host with kvm-85 kernel modules and > > > am using the built qemu-system-x86_64 from the kvm-85 source package: > > > > > > loaded kvm module (kvm-85) > > > QEMU PC emulator version 0.10.0 (kvm-85), Copyright (c) 2003-2008 Fabrice > > > Bellard > > > > > > > 2. Your host kernel is Fedora 11 Preview, that should be fine with device > > > > assignment at first(and let's solve it first, SRIOV the next step). > > > > > > Ok, yeah I will stick with the v2.6.29 fc11 kernel on the KVM host for > > > the momemt to get e1000e working. But I will start building a > > > v2.6.30-rc3 kernel again for my fc11 host kernel as I do need SR-IOV at > > > some point... :-) > > > > > > > 3. Your KVM version seems like kvm-85, you may provide some dmesg on host > > > > side(I think you didn't use the KVM come along with kernel). > > > > > > Ok, now within the KVM guest running v2.6.29.2, I see the following: > > > > > > [ 2.669243] e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k6 > > > [ 2.672931] e1000e: Copyright (c) 1999-2008 Intel Corporation. > > > [ 2.674932] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 > > > [ 2.675181] 8139too Fast Ethernet driver 0.9.28 > > > [ 2.676783] e1000e 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 > > > (level, high) -> IRQ 10 > > > [ 2.678143] e1000e 0000:00:05.0: setting latency timer to 64 > > > [ 2.679539] e1000e 0000:00:05.0: irq 24 for MSI/MSI-X > > > [ 2.679603] e1000e 0000:00:05.0: irq 25 for MSI/MSI-X > > > [ 2.679659] e1000e 0000:00:05.0: irq 26 for MSI/MSI-X > > > [ 2.698039] FDC 0 is a S82078B > > > [ 2.801673] 0000:00:05.0: eth0: (PCI Express:2.5GB/s:Width x1) > > > 00:e0:81:c0:90:b2 > > > [ 2.802811] 0000:00:05.0: eth0: Intel(R) PRO/1000 Network Connection > > > [ 2.803697] 0000:00:05.0: eth0: MAC: 3, PHY: 8, PBA No: ffffff-0ff > > > > > > And the folllowing from /proc/interrupts inside of the KVM guest: > > > > > > 24: 117 0 0 0 0 0 > > > 0 0 0 0 PCI-MSI-edge eth1-rx-0 25: > > > 0 0 0 0 0 0 0 > > > 0 0 0 PCI-MSI-edge eth1-tx-0 26: 2 > > > 0 0 0 0 0 0 > > > 0 0 0 PCI-MSI-edge eth1 > > > > > > ethtool eth1 reports that Link is detected, but I am still unable to get > > > a dhcp to work. > > > > It's a little strange that I checked all the log you posted, but can't find > > anything suspicious...(Except you got a MCE log in your dmesg, but I don't > > think it would relate to this). > > > > You also already have interrupts in the guest for eth1-rx-0 and eth1, so at > > least part of interrupts can be delivered to the guest. > > > > You can try to connect the port to another NIC port directly. Set fixed ip for > > each, then ping each other. > > > > You can also try to disable MSI-X capability in QEmu. Just using "#if > > 0/#endif" to wrap "#ifdef KVM_CAP_DEVICE_MSIX/#endif" in > > hw/assigned_device_pci_cap_init(). Then the device would use MSI. > > > > If I am lucky enough to find a 82574L card by hand, I would give it a try... > > > > -- > > regards > > Yang, Sheng > > > > Greetings Sheng, > > So I updated my FC11 Host to kernel v2.6.30-rc3 (and enabled ext4 of > course) and rebuilt the kvm-85 source kernel module and > qemu-system-x86_64 and I am now able to get dhcp and IP ops from the > 02:00.0 device on my IOH-5520 board with the KVM guest using a v2.6.29.2 > kernel!! Everything is looking good with the v2.6.29.2, but after a > quick reboot back into my v2.6.30-rc3 KVM guest kernel build e1000e it > looks like I am unable to get dhcp. > > Rebooting back into KVM Guest kernel v2.6.29.2 brings the pci-stub > assigned e1000e 82574L assigned with dhcp and everything looks > good! :-) > > I will keep poking at the v2.6.30-rc KVM guests (I am going to do a > complete rebuild) and see if it does not start move IP packets as well.. > > Thanks for all of your help in getting setup! > > --nab > > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
