I'm really stymied by this issue so thanks in advance for any help!
Problem statement:
The virtual functions of an SR-IOV NIC are not being assigned assigned as specified in the VM guest's XML definitions.
Data Points:
Server Information: HP ProLiant DL165 G7
Ethernet Card Information: Intel 82576
uname: Linux 3.0.0-16-server x86_64 GNU/Linux
OS Release: Ubuntu 11.10
KVM version: 0.14.1+noroms-0ubuntu6.2
Steps to Reproduce:
# ip link show:
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 02:22:25:00:00:00
vf 1 MAC 02:22:25:00:00:01
vf 2 MAC 02:22:25:00:00:02
vf 3 MAC 02:22:25:00:00:03
vf 4 MAC 02:22:25:00:00:04
vf 5 MAC 02:22:25:00:00:05
vf 6 MAC 02:22:25:00:00:06
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN qlen 1000
link/ether 00:1b:21:7d:70:16 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 02:22:25:01:00:00
vf 1 MAC 02:22:25:01:00:01
vf 2 MAC 02:22:25:01:00:02
vf 3 MAC 02:22:25:01:00:03
vf 4 MAC 02:22:25:01:00:04
vf 5 MAC 02:22:25:01:00:05
vf 6 MAC 02:22:25:01:00:06
# lspci |grep 05
05:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
05:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
# virsh nodedev-list |grep 05
pci_0000_05_00_0
pci_0000_05_00_1
pci_0000_05_10_0
pci_0000_05_10_1
pci_0000_05_10_2
pci_0000_05_10_3
pci_0000_05_10_4
pci_0000_05_10_5
pci_0000_05_10_6
pci_0000_05_10_7
pci_0000_05_11_0
pci_0000_05_11_1
pci_0000_05_11_2
pci_0000_05_11_3
pci_0000_05_11_4
pci_0000_05_11_5
# virsh nodedev-dettach pci_0000_05_10_0
Device pci_0000_05_10_0 dettached
# virsh nodedev-dettach pci_0000_05_10_1
Device pci_0000_05_10_1 dettached
# virsh nodedev-dettach pci_0000_05_11_0
Device pci_0000_05_11_0 dettached
# virsh nodedev-dettach pci_0000_05_11_1
Device pci_0000_05_11_1 dettached
# virsh nodedev-dumpxml pci_0000_05_10_0|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>16</slot>
<function>0</function>
# virsh nodedev-dumpxml pci_0000_05_10_1|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>16</slot>
<function>1</function>
# virsh nodedev-dumpxml pci_0000_05_11_0|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>17</slot>
<function>0</function>
# virsh nodedev-dumpxml pci_0000_05_11_1|grep -e bus -e slot -e fun |head -3
<bus>5</bus>
<slot>17</slot>
<function>1</function>
== HOST-A ==
XML:
...
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x10' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x11' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</hostdev>
...
dmesg:
[ 2.765683] Intel(R) Virtual Function Network Driver - version 1.0.8-k0
[ 2.765687] Copyright (c) 2009 - 2010 Intel Corporation.
[ 2.765809] igbvf 0000:00:04.0: setting latency timer to 64
[ 2.766050] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
[ 2.766073] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
[ 2.766094] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
[ 3.072509] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
[ 3.072514] igbvf 0000:00:04.0: Address: 02:22:25:00:00:00
[ 3.072516] igbvf 0000:00:04.0: MAC: 1
[ 3.072658] igbvf 0000:00:05.0: setting latency timer to 64
[ 3.072902] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
[ 3.072924] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
[ 3.072946] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
[ 3.241325] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
[ 3.241329] igbvf 0000:00:05.0: Address: 02:22:25:00:00:04
[ 3.241332] igbvf 0000:00:05.0: MAC: 1
[ 4.070069] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
[ 4.110094] igbvf 0000:00:05.0: Link is Up 1000 Mbps Full Duplex
[ 4.110127] igbvf 0000:00:04.0: Link is Up 1000 Mbps Full Duplex
ip link
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:22:25:00:00:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:22:25:00:00:04 brd ff:ff:ff:ff:ff:ff
== HOST-B ==
XML:
...
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x10' function='0x2'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x05' slot='0x11' function='0x2'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</hostdev>
...
dmesg:
[ 2.235455] igbvf 0000:00:04.0: setting latency timer to 64
[ 2.235682] igbvf 0000:00:04.0: irq 42 for MSI/MSI-X
[ 2.235704] igbvf 0000:00:04.0: irq 43 for MSI/MSI-X
[ 2.235724] igbvf 0000:00:04.0: irq 44 for MSI/MSI-X
[ 2.540856] igbvf 0000:00:04.0: Intel(R) 82576 Virtual Function
[ 2.540861] igbvf 0000:00:04.0: Address: 02:22:25:01:00:00
[ 2.540864] igbvf 0000:00:04.0: MAC: 1
[ 2.541017] igbvf 0000:00:05.0: setting latency timer to 64
[ 2.541272] igbvf 0000:00:05.0: irq 45 for MSI/MSI-X
[ 2.541294] igbvf 0000:00:05.0: irq 46 for MSI/MSI-X
[ 2.541314] igbvf 0000:00:05.0: irq 47 for MSI/MSI-X
[ 2.800795] igbvf 0000:00:05.0: Intel(R) 82576 Virtual Function
[ 2.800799] igbvf 0000:00:05.0: Address: 02:22:25:01:00:04
[ 2.800802] igbvf 0000:00:05.0: MAC: 1
ip link:
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 02:22:25:01:00:00 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 02:22:25:01:00:04 brd ff:ff:ff:ff:ff:ff
Expected Results:
HOST-A should have the following assignment:
* eth0 should = eth0-vf0 (bus:5, slot:10, fun:0), mac: 02:22:25:00:00:00
* eth1 should = eth1-vf0 (bus:5, slot:11, fun:0), mac: 02:22:25:01:00:00
HOST-b should have the following assignment:
* eth0 should = eth0-vf1 (bus:5, slot:10, fun:1), mac: 02:22:25:00:00:01
* eth1 should = eth1-vf1 (bus:5, slot:11, fun:1), mac: 02:22:25:01:00:01--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
