Hi, Mabye that's the problem ? : May 27 09:13:44 on-10-177-32-62 kernel: [140204.533733] ixgbe 0000:01:00.0 eth0: VF Reset msg received from vf 13 May 27 09:13:44 on-10-177-32-62 kernel: [140204.533970] ixgbe 0000:01:00.0: VF 13 has no MAC address assigned, you may have to assign one manually May 27 09:13:44 on-10-177-32-62 kernel: [140204.552220] ixgbe 0000:01:00.1 eth1: VF Reset msg received from vf 13 May 27 09:13:44 on-10-177-32-62 kernel: [140204.552228] ixgbe 0000:01:00.1: VF 13 has no MAC address assigned, you may have to assign one manually May 27 09:13:45 on-10-177-32-62 kernel: [140205.674551] ixgbe 0000:01:00.1 eth1: VF Reset msg received from vf 12 May 27 09:13:45 on-10-177-32-62 kernel: [140205.694783] pci-stub 0000:01:13.1: irq 288 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.726747] pci-stub 0000:01:13.1: irq 288 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.726766] pci-stub 0000:01:13.1: irq 289 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.786692] pci-stub 0000:01:13.1: irq 288 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.786702] pci-stub 0000:01:13.1: irq 289 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140205.786710] pci-stub 0000:01:13.1: irq 290 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.254300] ixgbe 0000:01:00.0 eth0: VF Reset msg received from vf 13 May 27 09:13:45 on-10-177-32-62 kernel: [140206.286346] pci-stub 0000:01:13.2: irq 407 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.306280] pci-stub 0000:01:13.2: irq 407 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.306293] pci-stub 0000:01:13.2: irq 408 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.354379] pci-stub 0000:01:13.2: irq 407 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.354393] pci-stub 0000:01:13.2: irq 408 for MSI/MSI-X May 27 09:13:45 on-10-177-32-62 kernel: [140206.354403] pci-stub 0000:01:13.2: irq 409 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.174615] ixgbe 0000:01:00.1 eth1: VF Reset msg received from vf 13 May 27 09:13:49 on-10-177-32-62 kernel: [140210.207755] pci-stub 0000:01:13.3: irq 410 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.251553] pci-stub 0000:01:13.3: irq 410 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.251568] pci-stub 0000:01:13.3: irq 411 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.331491] pci-stub 0000:01:13.3: irq 410 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.331506] pci-stub 0000:01:13.3: irq 411 for MSI/MSI-X May 27 09:13:49 on-10-177-32-62 kernel: [140210.331518] pci-stub 0000:01:13.3: irq 412 for MSI/MSI-X Regards Dominik 2013/5/26 Dominik Mostowiec <dominikmostowiec@xxxxxxxxx>: > Hi, > >> First - I just got confirmation from the libnl maintainer that the bug I >> mentioned causing problems with max_vfs > 50 *is* an issue in libnl >> 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and >> it should solve at least some of your problems. > > Upgrade libnl to version 3.2.22 helps. > Thanks. > >> 1) when you have max_vfs = 10, you can assign the devices and mac >> addresses are set properly, i.e. everything works. >> 2) when you have max_vfs = 35, you can assign the devices, but the mac >> addresses are not set properly. >> 3) when you have max_vfs = 63, you can't assign the devices because you >> get this error: >> >> internal error missing IFLA_VF_INFO in netlink response >> >> Is this correct? > > Yes. > Before upgrade libnl :-) > Upgrede helps only for limit max_vfs. > >> (BTW, it may get a bit confusing if you leave your networks named as >> "vnet0" and "vnet1" - although it's a different namespace, libvirt uses >> "vnetN" (where n is 0 - whatever) as the name for the tap device >> associated with each guest interface). > > You're right. > I changed this. > >> the output of "ip link show dev $PF". > Hmm, >> ip link show dev eth0 > Shows only: > 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq > master bond0 state UP qlen 1000 > link/ether b8:ca:3a:5b:a6:3a brd ff:ff:ff:ff:ff:ff > > I see VFs like network interfaces eth2-eth127. > When VF is attached to VM it disappear from list on host. > > -- > Regards > Dominik > > > 2013/5/24 Laine Stump <laine@xxxxxxxxx>: >> On 05/24/2013 09:16 AM, Dominik Mostowiec wrote: >> >>>>> Do you have this problem with mac addresses when max_vfs is set to an >>> even lower number? >> >> >> First - I just got confirmation from the libnl maintainer that the bug I >> mentioned causing problems with max_vfs > 50 *is* an issue in libnl >> 3.2.16 - it is fixed in libnl 3.2.22. Try to upgrade to that version and >> it should solve at least some of your problems. >> >> >>> >>> No, >>> When max_fs=10, macs in VM are OK. >> >> >> Interesting. >> >> So now I'm curious about something - can you try adding a vlan tag to >> your networks and see if the vlan tag is set properly when max_vfs is a >> low number (10 or 7): >> >> <network> >> <name>vnet0</name> >> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> >> <vlan> >> <tag id='42'/> >> </vlan> >> <forward mode='hostdev' managed='yes'> >> <pf dev='eth0'/> >> </forward> >> </network> >> >> >> (you will have to do "virsh net-destroy vnet0; virsh net-start vnet0" >> after making the modification to the network). If the vlan tag is >> successfully set, it will show up next to the mac address for the VF in >> the output of "ip link show dev $PF". >> >> (BTW, it may get a bit confusing if you leave your networks named as >> "vnet0" and "vnet1" - although it's a different namespace, libvirt uses >> "vnetN" (where n is 0 - whatever) as the name for the tap device >> associated with each guest interface). >> >> To summarize: >> >> 1) when you have max_vfs = 10, you can assign the devices and mac >> addresses are set properly, i.e. everything works. >> 2) when you have max_vfs = 35, you can assign the devices, but the mac >> addresses are not set properly. >> 3) when you have max_vfs = 63, you can't assign the devices because you >> get this error: >> >> internal error missing IFLA_VF_INFO in netlink response >> >> Is this correct? >> >>> >>> Phisical interfaces: >>>> ifconfig eth0 >>> eth0 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 >>> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 >> >> Okay, so that's not the problem. >> >>> >>>> ifconfig eth1 >>> eth1 Link encap:Ethernet HWaddr b8:ca:3a:5b:a6:38 >>> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 >>> >>>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag >>>>> for ixgbe vfs was not working properly (although it works fine for igb >>>>> devices) >>> Domain config: >>> <interface type="network"> >>> <alias name="hostdev0"/> >>> <source network="vnet0"/> >>> <mac address="52:54:0a:b1:48:fc"/> >>> </interface> >>> <interface type="network"> >>> <alias name="hostdev1"/> >>> <source network="vnet1"/> >>> <mac address="52:54:0a:b1:48:fc"/> >>> </interface> >>> >>> Networks defined: >>> <network> >>> <name>vnet0</name> >>> <uuid>ec49896a-a0b5-4944-a81f-9f0cdf578871</uuid> >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth0'/> >>> </forward> >>> </network> >>> >>> <network> >>> <name>vnet1</name> >>> <uuid>6c8319e8-8b53-4382-9f4e-2400819b00a9</uuid> >>> <forward mode='hostdev' managed='yes'> >>> <pf dev='eth1'/> >>> </forward> >>> </network> >>> >>> Regards >>> Dominik >>> >>> 2013/5/24 Laine Stump <laine@xxxxxxxxx>: >>>> On 05/24/2013 04:45 AM, Dominik Mostowiec wrote: >>>>> Hi, >>>>> I have libnl3, kernel 3.8.8, os ubuntu 12.04, net:intel10G. >>>>> Upgrade libnl to 3.2.16 not helps, >>>>> >>>>> I don't know - this is probably another problem, when I have set >>>>> max_vfs=35,35 VF is attaching to VM but mac in VM is not set >>>>> propertly. >>>> Do you have this problem with mac addresses when max_vfs is set to an >>>> even lower number? >>>> >>>> Ah, another thing - did you "ifconfig up" the PF before you started the >>>> guest? >>>>> Libvirt probably set this macs because in syslog: >>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171792] ixgbe >>>> >>>> Hmm. ixgbe. I just got a report yesterday that setting of the vlan tag >>>> for ixgbe vfs was not working properly (although it works fine for igb >>>> devices) >>>> >>>> >>>>> 0000:01:00.0: setting MAC 52:54:0a:b1:48:f7 on VF 0 >>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.171799] ixgbe >>>>> 0000:01:00.0: Reload the VF driver to make this change effective. >>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175054] ixgbe >>>>> 0000:01:00.1: setting MAC 52:54:0a:b1:48:f7 on VF 0 >>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.175060] ixgbe >>>>> 0000:01:00.1: Reload the VF driver to make this change effective. >>>>> May 23 14:48:59 on-10-177-32-62 kernel: [12574.307172] pci-stub >>>>> 0000:01:12.4: enabling device (0000 -> 0002) >>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.662890] assign device 0:1:12.4 >>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.665210] pci-stub >>>>> 0000:01:12.5: enabling device (0000 -> 0002) >>>>> May 23 14:49:00 on-10-177-32-62 kernel: [12575.765609] assign device 0:1:12.5 >>>>> But in VM i have another macs probably old from VF. >>>>> >>>>> --- >>>>> Regards >>>>> Dominik >>>>> >>>>> >>>>> >>>>> 2013/5/24 Laine Stump <laine@xxxxxxxxx>: >>>>>> On 05/22/2013 06:32 PM, Dominik Mostowiec wrote: >>>>>>> I tested on 1.0.5 patched version and vm with 2 vfs working fine. >>>>>>> >>>>>>> I have another problem >>>>>>> When i set max_vfs=63: >>>>>>> internal error missing IFLA_VF_INFO in netlink response >>>>>>> >>>>>>> Its working when max_vfs=31 >>>>>>> >>>>>> What is the platform, kernel version, and what version of libnl are you >>>>>> using (libnl 1 or libnl3?) >>>>>> >>>>>> There have been multiple bugs recently filed and fixed in this area on >>>>>> RHEL6/CentOS6. In particular, there was a problem with max_vfs > 50 that >>>>>> had exactly this symptom. It was solved by a patch to libnl-1.1. If your >>>>>> platform uses libnl-1.1, this could be the source of the problem. It is >>>>>> fixed in the upstream libnl-1.1.4 maintenance release. >>>>>> >>>>>> I unfortunately am not sure where the libnl-1.1 git is (I only have the >>>>>> git tree for libnl3), so I can't give you the exact commit message, but >>>>>> in short the problem was that libnl was using too small of a buffer for >>>>>> the netlink socket, so when there were a lot of vfs, the netlink message >>>>>> containing vf info was truncated. >>>>>> >>>>>> >>>>>>> -- >>>>>>> Dominik >>>>>>> >>>>>>> 21 maj 2013 15:19, "Dominik Mostowiec" <dominikmostowiec@xxxxxxxxx >>>>>>> <mailto:dominikmostowiec@xxxxxxxxx>> napisał(a): >>>>>>> >>>>>>> Hmm, >>>>>>> It seems to be working (after only simple tests). >>>>>>> >>>>>>> -- >>>>>>> Dominik >>>>>>> >>>>>>> >>>>>>> 2013/5/21 Ján Tomko <jtomko@xxxxxxxxxx <mailto:jtomko@xxxxxxxxxx>> >>>>>>> >>>>>>> On 05/21/2013 01:37 PM, Laine Stump wrote: >>>>>>> > On 05/21/2013 04:03 AM, Ján Tomko wrote: >>>>>>> >> On 05/21/2013 09:32 AM, Dominik Mostowiec wrote: >>>>>>> >>> hi, >>>>>>> >>> I try to add 2 VF by "hostdev". >>>>>>> >>> Networks (vnet0, vnet1) with: >>>>>>> >>> <forward mode='hostdev' managed='yes'> >>>>>>> >>> <pf dev='eth1'/> >>>>>>> >>> ..... >>>>>>> >>> >>>>>>> >>> Domain: >>>>>>> >>> <interface type="network"> >>>>>>> >>> <source network="vnet0"/> >>>>>>> >>> .... >>>>>>> >>> <interface type="network"> >>>>>>> >>> <source network="vnet1"/> >>>>>>> >>> .... >>>>>>> >>> >>>>>>> >>> virsh create error: >>>>>>> >>> "error: internal error process exited while connecting to >>>>>>> monitor: kvm: >>>>>>> >>> -device >>>>>>> pci-assign,configfd=25,host=01:10.1,id=hostdev0,bus=pci.0,addr=0x4: >>>>>>> >>> Duplicate ID 'hostdev0' for device" >>>>>>> >>> >>>>>>> >> Hi, >>>>>>> >> >>>>>>> >> it seems we have been assigning the same id to all network >>>>>>> hostdevs until this >>>>>>> >> recent commit (not yet released): >>>>>>> >> >>>>>>> >> http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=6597cc25 >>>>>>> > >>>>>>> > >>>>>>> > If that patch has such an effect, it's purely accidental. >>>>>>> Have you >>>>>>> > tested this? (I plan to test a before and after as soon as >>>>>>> I've had >>>>>>> > breakfast) >>>>>>> > >>>>>>> >>>>>>> I haven't tested it and looking at the code again, I might've >>>>>>> been wrong :( >>>>>>> >>>>>>> Sorry about that. >>>>>>> >>>>>>> Jan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pozdrawiam >>>>>>> Dominik >>>>>>> >>>>> >>> >>> >>> -- >>> Pozdrawiam >>> Dominik >>> >> > > > > -- > Pozdrawiam > Dominik -- Pozdrawiam Dominik -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list