On Fri, 2012-05-18 at 18:38 +0000, Shesha Sreenivasamurthy wrote: > Alex Williamson <alex.williamson <at> redhat.com> writes: > > > > > On Wed, 2012-05-16 at 17:49 +0000, Shesha Sreenivasamurthy wrote: > > > I'm using FreeBSD 9.0 (FreeBSD freebsd9-i386 9.0-RELEASE FreeBSD 9.0-RELEASE) > > > as my guest OS with qemu-kvm (qemu-kvm-1.1-rc2/x86_64-softmmu/ > > > qemu-system-x86_64) running on centos 6.2 (CentOS release 6.2) with > > > 2.6.32-220.el6.x86_64 kernel. I am using Intel's 82599EB 10-Gigabit hardware > > > on the host, which I am trying to export into the FreeBSD guest OS. The guest > > > driver (version 2.4.4) will claim it. However, the status is shown as > > > "no carrier" an therefore cannot ping any other system. > > > > > > $ ifconfig ix0 > > > ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > > > > > options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU, > > > VLAN_HWCSUM,TSO4,VLAN_HWTSO> > > > ether 90:e2:ba:0d:3d:b0 > > > inet 4.4.4.1 netmask 0xffffff00 broadcast 4.4.4.255 > > > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> > > > media: Ethernet autoselect > > > status: no carrier > > > > > > On the host side, before the ixgbe is loaded in the guest, /proc/interrupts > > > show the following entry for kvm > > > > > > # cat /proc/interrupts | grep kvm > > > 30: 0 0 0 0 0 0 > > > 0 0 0 0 0 0 0 > > > 0 0 0 IR-IO-APIC-fasteoi kvm_assigned_intx_device > > > > > > However, after the driver is loaded, the above entry is gone but two new > > > entries show up. > > > > > > # cat /proc/interrupts | grep kvm > > > 76: 0 0 0 0 0 0 > > > 0 0 0 0 0 0 0 > > > 0 0 0 IR-PCI-MSI-edge kvm_assigned_msix_device > > > 77: 0 0 0 0 0 0 > > > 0 0 0 0 0 0 0 > > > 0 0 0 IR-PCI-MSI-edge kvm_assigned_msix_device. > > > > This is expected, the driver enabled MSI-X. If you have an equivalent > > of lspci, you should be able to see before that MSI/MSI-X is disabled > > and after MSI-X is enabled with 2 vectors. Do you ever get interrupts > > on these lines? > > > > > To make sure, the my host configurations are fine, I loaded linux guest, with > > > exact same qemu params and host configurations. It works just fine. > > > > > > What I noticed in case linux guest is, to start with same entry is seen in > > > /proc/interrupts. However, after the ixgbe is loaded in the guest, I see only > > > one entry instead of two as in case of freebsd. > > > > > > # cat /proc/interrupts | grep kvm > > > 30: 0 0 0 0 0 0 > > > 0 0 0 0 0 0 0 > > > 0 0 0 IR-IO-APIC-fasteoi kvm_assigned_intx_device > > > > > > # cat /proc/interrupts | grep kvm > > > 76: 0 0 0 0 0 0 > > > 0 0 0 0 0 0 0 > > > 0 0 0 IR-PCI-MSI-edg kvm_assigned_msi_device > > > > Linux may be enabling MSI instead of MSI-X, that would be Ok. Check > > lspci in the guest to see what Linux thinks it's using. > > > > > Similar problem was reported earlier too: > > > http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806 > > > > > > Any ideas or advice will be highly valued > > > > Right, we fixed how we do MSI-X vector allocation so we can better track > > how this driver works. That allows an 82576 (igb) device to work, but > > there might still be a FreeBSD ixgbe driver issue that's not allowing it > > to see the link state. Thanks, > > > > Alex > > > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > the body of a message to majordomo <at> vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > I tried the same card on a natively running freebsd9.0 and used the same > drivers and it works fine. > > It was installed using the same ISO from which I created the vm disk image, > used the same driver and it works fine. I have captured the output of loading > ixgbe driver in a VM with VT-d and loading the same driver on freebsd running > natively on a system. --- vm 2012-05-18 12:44:44.168458069 -0600 +++ native 2012-05-18 12:45:01.288297371 -0600 @@ -1,7 +1,8 @@ ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4> port -0xc040-0xc05f mem 0xfea80000-0xfeafffff,0xfebd0000-0xfebd3fff irq 11 -at device 3.0 on pci0 -ix0: Using MSIX interrupts with 2 vectors +0xe880-0xe89f mem 0xf8e80000-0xf8efffff,0xf8e7c000-0xf8e7ffff irq 30 +at device 0.0 on pci3 +ix0: Using MSIX interrupts with 9 vectors The BAR addresses moved, the irq is different and the slot is different, all of that is expected. But why does the VM case use 2 vectors while the native case uses 9 vectors? Can you send lspci -vvv for the device from both the Linux host and from a Linux guest with the device assigned? We saw a Linux guest was using MSI instead of MSI-X, I wonder if it's because it has a better error path when it sees the wrong number of MSI-X vectors. Thanks, Alex +ix0: RX Descriptors exceed system mbuf max, using default instead! ixgbe_init_mac_link_ops_82599 ixgbe_identify_phy_generic ixgbe_identify_phy_generic: -17 @@ -26,7 +27,7 @@ Enter ixgbe_check_mac_link_generic LINKS orig: 2C080001, new: 2C080001 link_up_wait_to_complete: 0 -link_up: 1 +link_up: 0 IXGBE_LINK_SPEED_10GB_FULL: 0x80, IXGBE_LINK_SPEED_1GB_FULL: 0x20 IXGBE_LINK_SPEED_100_FULL: 0x8, IXGBE_LINK_SPEED_UNKNOWN: 0x0 speed: 0x80 @@ -48,7 +49,7 @@ multispeed_fiber: 0x1 reset_if_overtemp: 0x0 --------------- -ix0: Ethernet address: 90:e2:ba:0d:3d:b0 +ix0: Ethernet address: 90:e2:ba:0d:3e:5c Enter ixgbe_get_bus_info_generic link_status: 0x1082, bus_width: 0x0, bus_sped: 0x0 NEW link_status: 0x1082, bus_width: 0x8, bus_sped: 0x1388 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
