On Fri, 2012-03-23 at 11:59 -0400, Scott wrote: > I'm really stymied by this issue so thanks in advance for any help! > > Problem statement: > The virtual functions of an SR-IOV NIC are not being assigned assigned as specified in the VM guest's XML definitions. > > Data Points: > Server Information: HP ProLiant DL165 G7 > Ethernet Card Information: Intel 82576 > uname: Linux 3.0.0-16-server x86_64 GNU/Linux > OS Release: Ubuntu 11.10 > KVM version: 0.14.1+noroms-0ubuntu6.2 > > Steps to Reproduce: > > # ip link show: > 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000 > link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff > vf 0 MAC 02:22:25:00:00:00 > vf 1 MAC 02:22:25:00:00:01 > vf 2 MAC 02:22:25:00:00:02 > vf 3 MAC 02:22:25:00:00:03 > vf 4 MAC 02:22:25:00:00:04 > vf 5 MAC 02:22:25:00:00:05 > vf 6 MAC 02:22:25:00:00:06 > 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000 > link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff > vf 0 MAC 02:22:25:01:00:00 > vf 1 MAC 02:22:25:01:00:01 > vf 2 MAC 02:22:25:01:00:02 > vf 3 MAC 02:22:25:01:00:03 > vf 4 MAC 02:22:25:01:00:04 > vf 5 MAC 02:22:25:01:00:05 > vf 6 MAC 02:22:25:01:00:06 > > # lspci |grep 05 > 05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) > 05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) > 05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > 05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) > > # virsh nodedev-list |grep 05 > pci_0000_05_00_0 > pci_0000_05_00_1 > pci_0000_05_10_0 > pci_0000_05_10_1 > pci_0000_05_10_2 > pci_0000_05_10_3 > pci_0000_05_10_4 > pci_0000_05_10_5 > pci_0000_05_10_6 > pci_0000_05_10_7 > pci_0000_05_11_0 > pci_0000_05_11_1 > pci_0000_05_11_2 > pci_0000_05_11_3 > pci_0000_05_11_4 > pci_0000_05_11_5 These are mapped as: pci_0000_05_00_0 -> eth0 pci_0000_05_00_1 -> eth1 pci_0000_05_10_0 -> eth0-vf0 pci_0000_05_10_1 -> eth1-vf0 pci_0000_05_10_2 -> eth0-vf1 pci_0000_05_10_3 -> eth1-vf1 pci_0000_05_10_4 -> eth0-vf2 pci_0000_05_10_5 -> eth1-vf2 pci_0000_05_10_6 -> eth0-vf3 pci_0000_05_10_7 -> eth1-vf3 pci_0000_05_11_0 -> eth0-vf4 pci_0000_05_11_1 -> eth1-vf4 pci_0000_05_11_2 -> eth0-vf5 pci_0000_05_11_3 -> eth1-vf5 pci_0000_05_11_4 -> eth0-vf6 pci_0000_05_11_5 -> eth1-vf6 > # virsh nodedev-dettach pci_0000_05_10_0 > Device pci_0000_05_10_0 dettached > > # virsh nodedev-dettach pci_0000_05_10_1 > Device pci_0000_05_10_1 dettached > > # virsh nodedev-dettach pci_0000_05_11_0 > Device pci_0000_05_11_0 dettached > > # virsh nodedev-dettach pci_0000_05_11_1 > Device pci_0000_05_11_1 dettached > > # virsh nodedev-dumpxml pci_0000_05_10_0|grep -e bus -e slot -e fun |head -3 > <bus>5</bus> > <slot>16</slot> > <function>0</function> > > # virsh nodedev-dumpxml pci_0000_05_10_1|grep -e bus -e slot -e fun |head -3 > <bus>5</bus> > <slot>16</slot> > <function>1</function> > > # virsh nodedev-dumpxml pci_0000_05_11_0|grep -e bus -e slot -e fun |head -3 > <bus>5</bus> > <slot>17</slot> > <function>0</function> > > # virsh nodedev-dumpxml pci_0000_05_11_1|grep -e bus -e slot -e fun |head -3 > <bus>5</bus> > <slot>17</slot> > <function>1</function> > > == HOST-A == > > XML: > ... > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> > </hostdev> eth0-vf0 (02:22:25:00:00:00) > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x05' slot='0x11' function='0x0'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> > </hostdev> eth0-vf4 (02:22:25:00:00:04) > > dmesg: > [ 2.765683] Intel(R) Virtual Function Network Driver - version 1.0.8-k0 > [ 2.765687] Copyright (c) 2009 - 2010 Intel Corporation. > [ 2.765809] igbvf 0000:00:04.0: setting latency timer to 64 > [ 2.766050] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X > [ 2.766073] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X > [ 2.766094] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X > [ 3.072509] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function > [ 3.072514] igbvf 0000:00:04.0: Address: 02:22:25:00:00:00 > [ 3.072516] igbvf 0000:00:04.0: MAC: 1 > [ 3.072658] igbvf 0000:00:05.0: setting latency timer to 64 > [ 3.072902] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X > [ 3.072924] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X > [ 3.072946] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X > [ 3.241325] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function > [ 3.241329] igbvf 0000:00:05.0: Address: 02:22:25:00:00:04 > [ 3.241332] igbvf 0000:00:05.0: MAC: 1 > [ 4.070069] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex > [ 4.110094] igbvf 0000:00:05.0: Link is Up 1000 Mbps Full Duplex > [ 4.110127] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex > > ip link > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 > link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff > 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 > link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff Looks right > == HOST-B == > > XML: > ... > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x05' slot='0x10' function='0x2'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> > </hostdev> eth0-vf1 (02:22:25:00:00:01) > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x05' slot='0x11' function='0x2'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> > </hostdev> eth0-vf5 (02:22:25:00:00:05) > dmesg: > [ 2.235455] igbvf 0000:00:04.0: setting latency timer to 64 > [ 2.235682] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X > [ 2.235704] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X > [ 2.235724] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X > [ 2.540856] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function > [ 2.540861] igbvf 0000:00:04.0: Address: 02:22:25:01:00:00 > [ 2.540864] igbvf 0000:00:04.0: MAC: 1 > [ 2.541017] igbvf 0000:00:05.0: setting latency timer to 64 > [ 2.541272] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X > [ 2.541294] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X > [ 2.541314] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X > [ 2.800795] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function > [ 2.800799] igbvf 0000:00:05.0: Address: 02:22:25:01:00:04 > [ 2.800802] igbvf 0000:00:05.0: MAC: 1 > > ip link: > 2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000 > link/ether 02:22:25:01:00:00 brd ff:ff:ff:ff:ff:ff > 3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000 > link/ether 02:22:25:01:00:04 brd ff:ff:ff:ff:ff:ff Um... not expected > Expected Results: > HOST-A should have the following assignment: > * eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00 > * eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00 Nope, eth1-vf0 would be 5:10.1. The even VF come from PF func 0, odd from odd. You can unbind 5:00.1 from igb to see this, all the odd VFs will go away. You can also find the VFs linked under the PF in /sys/bus/pci/devices/0000:05:00.[01]. > HOST-b should have the following assignment: > * eth0 should = eth0-vf1 (bus:5, slot:10, fun:1), mac: 02:22:25:00:00:01 > * eth1 should = eth1-vf1 (bus:5, slot:11, fun:1), mac: 02:22:25:01:00:01-- Wait a sec, you originally did a nodedev-dumpxml for 5:10.[01] and 5:11.[01]. 5:10.0 & 5:11.0 are used for HOST-A. The xmldump snippet for HOST-B is using 5:10.2 & 5:11.2. Since you reference .1 again here, I can only assume that means HOST-B is actually using 5:10.1 & 5:11.1 (aka eth1-vf0 & eth1-vf4), which means the guest output is correct and expected. Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
