> -----Original Message----- > From: sendmail [mailto:justsendmailnothingelse@xxxxxxxxx] On Behalf Of > Laine Stump > Sent: Wednesday, December 16, 2015 12:31 AM > To: Libvirt <libvir-list@xxxxxxxxxx> > Cc: vyasevic@xxxxxxxxxx; Moshe Levi <moshele@xxxxxxxxxxxx> > Subject: Re: <interface type='hostdev'>vf configuration cleanup when > VM is delete > > On 12/15/2015 04:12 PM, Vlad Yasevich wrote: > > On 12/15/2015 02:45 PM, Laine Stump wrote: > >> On 12/15/2015 01:34 PM, Laine Stump wrote: > >>> On 12/13/2015 10:51 AM, Moshe Levi wrote: > >>>> Hi, > >>>> > >>>> I have a setup with libvirt 1.3.0 and OpenStack trunk. > >>>> > >>>> Before launched the VM ip link command show the following VF > >>>> mac/vlan configuration [1] > >>>> > >>>> When I launch a VM with <interface type='hostdev'> via openstack > >>>> api (OpenStack direct > >>>> port) > >>>> > >>>> I can see that the VF get the mac/vlan according to libvrit xml [2] > >>>> and ip link command [3], but when I delete the VM the mac/vlan > >>>> config are still shown as in [3] and not restored to [1] > >>>> > >>>> Shouldn’t libvirt restore the mac/vlan to [1]. > >>>> > >>>> The same problem exists when using <interface type='direct'> > >>>> (OpenStack macvtap port) but just for the MAC configuration of the VF. > >>>> > >>> What libvirt does is to restore the MAC address to whatever it was > >>> before we set it up for use with a guest. Although there are some > >>> sriov net drivers that (for some unfathomable reason) think it's > >>> cool to assign a random MAC address to each VF at boot time, the > "normal" thing is for the VFs to have a MAC address of all 0's to start with. > >>> So libvirt should be saving 00:00:00:00:00:00 (it will be in the > >>> file > >>> /var/run/libvirt/hostdevmgr/$ifname_vf$vfnum) then setting the MAC > >>> to use; when done, libvirt will read the 00:00:00:00:00:00 and use > >>> netlink to set the MAC address, but this is apparently failing. > >>> > >>> I checked on my Fedora 22 system with the igb driver, and found that > >>> if the MAC address was originally set to something other than 0's, > >>> it was restored properly by libvirt, but if it was set to all 0's originally, the > attempt to set it back to 0 would fail. > >>> > >>> I then tried doing the same thing with the "ip" utility: > >>> > >>> # ip link set dev p4p2 vf 0 mac 00:00:00:00:00:00 > >>> > >>> and I get the following response: > >>> > >>> RTNETLINK answers: Invalid argument > >>> > >>> So it appears that either the kernel or the NIC driver is refusing > >>> to set the MAC address to all 0's. I'm reasonably certain this is a > >>> regression in the kernel, > >> Sigh. It appears that this has "always" been the case - I just > >> checked on a 2.6.32-573 RHEL kernel, and a 3.10.x RHEL7.2 kernel, and > >> 4.1 (Fedora 22) and both of them also refuse to set the MAC address > >> to 00:00:00:00:00:00. I'm not sure if this limitation is in the NIC driver or > some basic code in the kernel. > > It's considered to be an invalid address by is_valid_ether_addr() function. > > > > There appear to be different behaviour in some adapters. In current > > upstream, it looks like a quarter of the VF capable drivers (bnxt, > > enic, fm10k, sfc) permit VF mac setting of all zeros. The others > > simply use is_valid_ether_addr function without specifically testing > > for all-0. I am not sure if this is HW related or simply oversights... Might > want to bringing this up on netdev. > > Thanks, Vlad! > > > Moshe, > > It sounds like in your case it is caused by code in the mlx driver, so hopefully > you can have some influence there. My path is a bit more difficult, as the > failure on mine is in the igb driver :-) Sure I will take it with Mellanox Driver team Ok, just saw the latest thread mail you can discard my previous one > > Sending a message to netdev is a good idea. It would be wonderful if all the > vendors could agree to: > > 1) initialize all VFs with a MAC address of 0 > 2) allow setting VF MAC address to 0 at any time. > > I'll put that on my to-do list :-P > > > > -vlad > > > >> > >>> although I can't say how long it's been there, as I don't normally > >>> pay attention to this (and as I said, many SRIOV NIC drivers don't > >>> default their VFs to 0 MAC addresses) > >>> > >>> What distro and kernel are you using for your tests? > >>> > >>> > >>>> [1] - 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > >>>> qdisc mq master ovs-system state UP mode DEFAULT group default qlen > >>>> 1000 > >>>> > >>>> link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff > >>>> > >>>> vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state > >>>> auto > >>>> > >>>> vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state > >>>> auto > >>>> > >>>> vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state > >>>> auto > >>>> > >>>> vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state > >>>> auto > >>>> > >>>> [2] - <interface type='hostdev' managed='yes'> > >>>> > >>>> <mac address=' fa:16:3e:11:af:fe '/> > >>>> > >>>> <driver name='kvm'/> > >>>> > >>>> <source> > >>>> > >>>> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' > >>>> function='0x7'/> > >>>> > >>>> </source> > >>>> > >>>> <vlan> > >>>> > >>>> <tag id='190'/> > >>>> > >>>> </vlan> > >>>> > >>>> <alias name='hostdev0'/> > >>>> > >>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > >>>> function='0x0'/> > >>>> > >>>> </interface> > >>>> > >>>> [3] 24: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > qdisc > >>>> mq master ovs-system state UP mode DEFAULT group default qlen 1000 > >>>> > >>>> link/ether e4:1d:2d:a5:f1:22 brd ff:ff:ff:ff:ff:ff > >>>> > >>>> vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state > >>>> auto > >>>> > >>>> vf 1 MAC 00:00:00:00:00:00, spoof checking off, link-state > >>>> auto > >>>> > >>>> vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state > >>>> auto > >>>> > >>>> vf 3 MAC fa:16:3e:11:af:fe, vlan 190, spoof checking off, > >>>> link-state enable > >>>> > >>>> > >>>> > >>>> -- > >>>> libvir-list mailing list > >>>> libvir-list@xxxxxxxxxx > >>>> https://www.redhat.com/mailman/listinfo/libvir-list > >>> F15 > >>> > >>> > >>> -- > >>> libvir-list mailing list > >>> libvir-list@xxxxxxxxxx > >>> https://www.redhat.com/mailman/listinfo/libvir-list > >> > > -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list