On 8/27/20 4:12 AM, Philipp Rosenberger wrote:
Hi,
I managed to get SR-IOV with an Intel I350 NIC to work.For this I
followed the documentation on this page:
https://wiki.libvirt.org/page/Networking#Assignment_from_a_pool_of_SRIOV_VFs_in_a_libvirt_.3Cnetwork.3E_definition
But as I have more VMs then VF on the NIC I also have a bridge wich
serves the other guests. As I run Debian Buster as host I followed the
documentation here:
https://wiki.libvirt.org/page/Networking#Debian.2FUbuntu_Bridging
Do I correctly assume that you're attaching the PF of the SRIOV card to
the bridge? (in particular, the PF of the same port that the VFs are from)
If I use only the SRIOV everything works as expected. All guests can be
reached for the network and the guests and host can reach each other.
The same goes for a sole bridged environment.
I think you missed a paragraph here - I'm inferring that at this point
you meant to say "But when I have both guests connected with an assigned
VF and guests connected via an virtio-net device connected to the bridge
via a tap device, the VF-connect guests cannot communicate with the
bridge-connected guests." Is that correct, or am I inferring too much?
As I dived into the issue I found an answer form the intel community:
https://community.intel.com/t5/Ethernet-Products/82599-VF-to-Linux-host-bridge/td-p/351802
By default all the ports on a Linux host bridge should have flood and
learning turned on, and I would have thought that (if manually adding a
mac address to the linux host bridge is enough to make the traffic flow)
that having flood+learning turnes on would be enough to get the bridge
to learn the proper port for traffic destined to the VF
Really, my first assumption would have been that the switch screwing up
was the switch in the SRIOV card incorrectly sending traffic for the
bridged guests directly out the PF's physical port instead of rather
than the Linux host bridge, and that the solution to make everything
work would be to somehow add the MAC address of the *bridged* guests
into the fdb of the SRIOV card.
[at this point I read further through your message and follow the link
to the slides...]
Ooohh..... from slide 33, I see that *is* what's being done. It sounds
like that's what you *are* doing - adding an fdb entry to the internal
switch in the SRIOV card, *not* to the Linux host bridge (as I had
thought right up until 3 paragraphs ago), correct?
It again is interesting though - makes it sound like the switch in the
SRIOV card has no learning and no flood.
It says I need to add the "VF mac addresses and eth0 mac address to
bridge forwarding database".
That definitely is what's said in the comments from the Intel forum. But
isn't that the opposite of what they're saying on slide 33 of the
presentation you linked to? My understanding is that it's saying that
you need to add the *bridged* guest interface's MAC address to the fdb
on the SRIOV card's switch.
I have done this with the following command:
bridge fdb add 52:54:00:3c:1c:e6 dev eth_lan0
The mac address is from my VM which is on the bridge. And the eth_lan0
interface is the physical interface of my bridge and also the PF of my
I350 NIC.
Ah, okay, this verifies my first assumption (that the bridge is attached
to the PF). It also verifying that you did what's suggested in slide 33
of the presentation, not what was suggested in the Intel forum post. Is
that correct?
This seems to work. But doing this manually is annoying and a bit of a
hassle when creating new VMs on the bridge.
I there a way to let libvirt do this work?
Well, if you create a libvirt network for your bridge device, something
like this:
<network>
<name>bridgenet</name>
<bridge name='br0'/>
<forward mode='bridge'/>
</network>
(this is an "unmanaged" network, i.e. libvirt expects the bridge to
already exist, doesn't add any iptables rules or dnsmasq instance - it
just creates tap devices and connects them to the already-existing
bridge), and then configure all your guest interfaces with:
<interface type='network'>
<source network='bridgenet'/>
...
</interface>
then a network hook will be called each time one of these interfaces is
added or removed, and that script can add/remove the fdb entry. Hooks
are described here:
https://libvirt.org/hooks.html
If you have an executable file named /etc/libvirt/hooks/network (or a
file of your own naming in /etc/libvirt/hooks/network.d if your libvirt
is 6.5.0 or newer) it will be called anytime a guest interface is added
or removed from a libvirt network. The arguments will describe the
current action, and stdin of the script will receive the full XML config
of the network, as well as the xml for a <networkport>, which contains
the interface's MAC address among other things. This should be enough
information to derive the proper "bridge fdb add" command. (you'll want
a similar clause in the same hook script that takes action when the
interface is removed from the network).
If you have trouble with the script, you can look in the #virt channel
on irc.oftc.net - if it's a weekday and the right time of day, you may
get immediate help (I'm actually curious to hear how this works out. It
*almost* sounds like something that could be integrated into the
standard config of libvirt, although I'm not sure how to make it easily
consumable).
(NB: you could also use the network hook script to perform some action
when the VFs are added/removed from guests).
Best regards
Philipp
PS:
This presentation shows it pretty well on page 33:
https://events.static.linuxfound.org/sites/events/files/slides/LinuxConJapan2014_makita_0.pdf
