Re: Sr-iov passthrough - no packet arrive to guest

"Yaron Illouz" <yaroni@xxxxxxxxxx> · Thu, 29 Jan 2015 01:27:23 +0200

I read a bit more
about the card that i am using  and

The problem was that
it filtered packet because it a trace that i replay and not real traffic

So i changed the destination
mac adress in the pcap to be the same as vm and all packet got into the vm.

From: libvirt-users-bounces@xxxxxxxxxx
[mailto:libvirt-users-bounces@xxxxxxxxxx] On
Behalf Of Yaron Illouz

Sent: Wednesday, January 28, 2015
6:06 PM

To: libvirt-users@xxxxxxxxxx

Subject: Re: 
Sr-iov passthrough - no packet arrive to guest

I can see from different post that if
working with sr-iov, i should work with vlan

Is this an obligation to work with vlan if
working with sr-iov?

If not according to which parameter will
the different vf get the traffic.

Let's say i declare max_vfs=7, how will
the traffic be seperated between the vm?

However till i get an answer i tried to
work with vlan

And i still don't get traffic in vm, but i
see packet arriving  in the physical interface of the host

I tried to do the following to define the
vlan

In virsh edit

<interface type='hostdev'
managed='yes'>

      <mac
address='52:54:00:60:7f:57'/>

<source>

<address type='pci' domain='0x0000' bus='0x15' slot='0x10'
function='0x0'/>

</source>

<vlan>

<tag id='44'/>

</vlan>

      <alias
name='hostdev0'/>

      <address
type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </interface>

In host ip link

23: eth6:
<BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
1000

    link/ether
38:ea:a7:8f:fa:98 brd ff:ff:ff:ff:ff:ff

    vf 0 MAC
52:54:00:60:7f:57, vlan 44

In guest

modprob 8021q

eth1      Link
encap:Ethernet  HWaddr 52:54:00:60:7F:57

      inet6 addr: fe80::5054:ff:fe60:7f57/64
Scope:Link

      UP BROADCAST RUNNING PROMISC MULTICAST 
MTU:1500  Metric:1

      RX packets:0 errors:0 dropped:0 overruns:0
frame:0

      TX packets:14 errors:0 dropped:0 overruns:0
carrier:0

      collisions:0 txqueuelen:1000

      RX bytes:0 (0.0 b)  TX bytes:1156 (1.1 KiB)

eth1.44   Link
encap:Ethernet  HWaddr 52:54:00:60:7F:57

      inet6 addr: fe80::5054:ff:fe60:7f57/64
Scope:Link

      UP BROADCAST RUNNING MULTICAST 
MTU:1500  Metric:1

      RX packets:0 errors:0 dropped:0 overruns:0
frame:0

      TX packets:7 errors:0 dropped:0 overruns:0
carrier:0

      collisions:0 txqueuelen:0

From: Yaron Illouz 

Sent: Tuesday, January 27, 2015
9:56 PM

To: 'libvirt-users@xxxxxxxxxx'

Subject: Sr-iov passthrough - no packet
arrive to guest

Hi 

I
am trying to do passthrough to guest using sriov and libvirt

I
tried to use both techniques from here

http://wiki.libvirt.org/page/Networking#Assignment_with_.3Chostdev.3E

Assignment from a pool of SRIOV VFs in a libvirt <network>
definition

Assignment with <interface type='hostdev'> (SRIOV
devices only)

<network>

   <name>passthrough-eth6</name>

   <forward mode='hostdev' managed='yes'>

     <pf dev='eth6'/>

   </forward>

 </network>

I
launch the vm, there is a vm with virt-install …. --network
network=passthrough-eth6 

I
inject traffic with a generator so I defined it in promisc mode without an ip address.(ifconfig
eth6 promisc up)

In my physical
device eth6 i can see RX increasing. But in the guest side nothing arrive in
the RX? Why?

I don’t know
in which log to look at to understand the problem

I
defined the  network-script/eth1 file in the guest is defined as static
(not dhcp) as the interface it is connected to in host have no ip.

I
have a another interface on the host connected via ovs and libvirt and it is
working fine – you can see it in log as vnet0(thanks to  this
mailing-list help).

In the guest part I attached the log but I think interesting
part is 

Jan 27 17:03:21 localhost kernel: ixgbevf 0000:00:04.0: NIC
Link is Up, 10 Gbps

I also attached dmesg output from guest

Here is the log from the host side, I can’t see an
error. /var/log/message     

Jan 27 12:02:29 1235 avahi-daemon[2598]: Withdrawing address
record for fe80::fc54:ff:fe92:7352 on vnet0.

Jan 27 12:02:29 1235 kernel: device vnet0 left promiscuous
mode

Jan 27 12:02:30 1235 kernel: ixgbe 0000:15:00.0: setting MAC
a2:fc:a8:89:2a:8f on VF 0

Jan 27 12:02:30 1235 ovs-vsctl: 00001|vsctl|INFO|Called as
ovs-vsctl --timeout=5 -- --if-exists del-port vnet0

Jan 27 12:02:30 1235 ovs-vswitchd: 00180|bridge|ERR|bridge
virbr0: mirror mirror-1-to-3 does not specify output; ignoring

Jan 27 12:02:31 1235 ntpd[2905]: Deleting interface #28
vnet0, fe80::fc54:ff:fe92:7352#123, interface stats: received=0, sent=0,
dropped=0, active_time=981 secs

Jan 27 12:02:31 1235 ntpd[2905]: peers refreshed

Jan 27 12:02:43 1235 kernel: ixgbe 0000:15:00.0: Reload the
VF driver to make this change effective.

Jan 27 12:02:43 1235 kernel: kvm: 7957: cpu0 unhandled
rdmsr: 0x345

Jan 27 12:02:43 1235 kernel: kvm: 7957: cpu0 unhandled
wrmsr: 0x680 data 0

…

Jan 27 12:02:51 1235 ovs-vsctl: 00001|vsctl|INFO|Called as
ovs-vsctl --timeout=5 -- --may-exist add-port br3 vnet0 -- set Interface vnet0
"external-ids:attached-mac=\"52:54:00:43:BF:8C\"" -- set
Interface vnet0
"external-ids:iface-id=\"309ed431-417f-2843-89b8-179a066aa595\""
-- set Interface vnet0 "external-ids:vm-id=\"870e3b65-953b-01ee-84a3-c2d05300c2d9\""
-- set Interface vnet0 external-ids:iface-status=active

Jan 27 12:02:51 1235 ovs-vswitchd: 00183|bridge|ERR|bridge
virbr0: mirror mirror-1-to-3 does not specify output; ignoring

Jan 27 12:02:51 1235 kernel: device vnet0 entered
promiscuous mode

Jan 27 12:02:51 1235 kernel: ixgbe 0000:15:00.0: setting MAC
52:54:00:d9:f1:bd on VF 0

Jan 27 12:02:51 1235 kernel: ixgbe 0000:15:00.0: Reload the
VF driver to make this change effective.

Jan 27 12:02:51 1235 kernel: pci-stub 0000:15:10.0: enabling
device (0000 -> 0002)

Jan 27 12:02:52 1235 avahi-daemon[2598]: Registering new
address record for fe80::fc54:ff:fe43:bf8c on vnet0.*.

Jan 27 12:02:54 1235 ntpd[2905]: Listen normally on 29 vnet0
fe80::fc54:ff:fe43:bf8c UDP 123

Jan 27 12:02:54 1235 ntpd[2905]: peers refreshed

Jan 27 12:03:17 1235 kernel: __ratelimit: 23 callbacks
suppressed

Jan 27 12:03:17 1235 kernel: kvm: 8006: cpu0 disabled
perfctr wrmsr: 0xc1 data 0xabcd

Jan 27 12:03:20 1235 kernel: ixgbe 0000:15:00.0: eth6: VF
Reset msg received from vf 0

And this is the /var/log/libvirt/qemu/probe8.log from the
host

2015-01-27 17:02:51.762+0000: starting up

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin
QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name probe8 -S -M rhel6.6.0
-enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1
-uuid 870e3b65-953b-01ee-84a3-c2d05300c2d9 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/probe8.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
-drive
file=/data/KVM/comp_VNF_MaveriQ_Probe2-cloud8.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive
file=/var/lib/libvirt/images/configuration8.iso,if=none,id=drive-virtio-disk0,format=raw,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0
-netdev tap,fd=25,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:43:bf:8c,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
-vnc 0.0.0.0:0 -vga cirrus -device pci-assign,host=15:10.0,id=hostdev0,configfd=26,bus=pci.0,addr=0x4
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

char device redirected to /dev/pts/1

I can see that virtual function succeeded because I got the
following output

virsh net-dumpxml  passthrough-eth6

<network connections='1'>

  <name>passthrough-eth6</name>

<uuid>f8fd37cd-3215-e4b0-17b1-0b5515140db9</uuid>

  <forward mode='hostdev' managed='yes'>

    <pf dev='eth6'/>

    <address type='pci' domain='0x0000'
bus='0x15' slot='0x10' function='0x0'/>

    <address type='pci' domain='0x0000'
bus='0x15' slot='0x10' function='0x2'/>

    <address type='pci' domain='0x0000'
bus='0x15' slot='0x10' function='0x4'/>

    <address type='pci' domain='0x0000'
bus='0x15' slot='0x10' function='0x6'/>

…

  </forward>

</network>

lspci | grep Intel

15:00.0 Ethernet controller: Intel Corporation 82599ES
10-Gigabit SFI/SFP+ Network Connection (rev 01)

15:00.1 Ethernet controller: Intel Corporation 82599ES
10-Gigabit SFI/SFP+ Network Connection (rev 01)

15:10.0 Ethernet controller: Intel Corporation 82599
Ethernet Controller Virtual Function (rev 01)

15:10.1 Ethernet controller: Intel Corporation 82599
Ethernet Controller Virtual Function (rev 01)

….

And Intel-IOMMU: enabled in the dmesg

From host dmesg

ixgbe 0000:15:00.0: setting MAC
52:54:00:97:c6:86 on VF 0