SR-IOV: no traffic isolation between VFs with Broadcom 10Gbps cards

[Yoann Juet <yoann.juet@xxxxxxxxxxxxxx>]

Hi all,

I'm testing on debian/unstable SR-IOV feature with Broadcom BCM57810 
cards and KVM hypervisor:

Compiled against library: libvirt 1.2.1
Using library: libvirt 1.2.1
Using API: QEMU 1.2.1
Running hypervisor: QEMU 1.7.0

bnx2x
-> firmware 7.8.17
-> driver from kernel 3.12.7

8 VFs are created on the first PF. For each VF, a specific mac address 
is set manually using "ip link set eth0 vf x mac xx:xx:xx:xx:xx" 
command. I run several KVM guests with PCI passthrough (same kernel, 
bnx2x driver and firmware as the host), performance is close to bare metal.

Well, that sounds good, until I start capturing the traffic inside each 
VM: host traffic is visible as well as traffic destined to other VM. 
It's like if internal card switching was inoperable. I made several 
tests with different kernels, different PCIe Passthrough method 
assignments for libvirt. All failed.

Has anyone successfully experiment SR-IOV with Broadcom cards on linux ?

-----

Some details:

01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet (rev 10)

01:09.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.2 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.3 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.4 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.5 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.6 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function
01:09.7 Ethernet controller: Broadcom Corporation NetXtreme II BCM57810 
10 Gigabit Ethernet Virtual Function


# virsh nodedev-dumpxml pci_0000_01_09_0
<device>
  <name>pci_0000_01_09_0</name>
  <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:09.0</path>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>1</bus>
    <slot>9</slot>
    <function>0</function>
    <product id='0x16af'>NetXtreme II BCM57810 10 Gigabit Ethernet 
Virtual Function</product>
    <vendor id='0x14e4'>Broadcom Corporation</vendor>
    <capability type='phys_function'>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </capability>
    <iommuGroup number='35'>
      <address domain='0x0000' bus='0x01' slot='0x09' function='0x0'/>
    </iommuGroup>
  </capability>
</device>


# virsh nodedev-dumpxml pci_0000_01_09_1
<device>
  <name>pci_0000_01_09_1</name>
  <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:09.1</path>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>1</bus>
    <slot>9</slot>
    <function>1</function>
    <product id='0x16af'>NetXtreme II BCM57810 10 Gigabit Ethernet 
Virtual Function</product>
    <vendor id='0x14e4'>Broadcom Corporation</vendor>
    <capability type='phys_function'>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </capability>
    <iommuGroup number='36'>
      <address domain='0x0000' bus='0x01' slot='0x09' function='0x1'/>
    </iommuGroup>
  </capability>
</device>


Guest A XML:
	...
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x09' function='0x0'/>
      </source>
    </hostdev>
	...


Guest B XML:

	...
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x09' function='0x1'/>
      </source>
    </hostdev>
	...


--
Université de Nantes - Direction des Systèmes d'Information

begin:vcard
fn:Yoann Juet
n:Juet;Yoann
org;quoted-printable;quoted-printable:Direction des Syst=C3=A8mes d'Information;P=C3=B4le R=C3=A9seau
adr;quoted-printable:BP 92208;;2 rue de la Houssini=C3=A8re;Nantes Cedex 3;;44322;France
email;internet:yoann.juet@xxxxxxxxxxxxxx
tel;work:02.53.48.49.26
tel;fax:02.53.48.49.09
tel;cell:06.73.15.42.19
version:2.1
end:vcard

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
libvirt-users mailing list
libvirt-users@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvirt-users