Greetings everybody, I am running stock ubuntu-natty 2.6.38-8, with KVM 0.14.0 and libvirt 0.8.8. I am using Intel 82599EB dual-port 10Gb NIC with VFs spawned by each one of the PFs. Those VFs are further assigned to KVM instances with PCI-passthrough using libvirt. Each instance receives 4 VFs (two VFs from each PF). Usually when KVM starts, messages like these appear in the kernel log of the physical machine: syslog.ubuntu-sata-51 146907 Mar 25 18:19:51 ubuntu-sata-51 kernel: [51267.029108] pci-stub 0000:03:13.1: claimed by stub syslog.ubuntu-sata-51 146908 Mar 25 18:19:51 ubuntu-sata-51 kernel: [51267.398569] pci-stub 0000:03:12.3: claimed by stub syslog.ubuntu-sata-51 146909 Mar 25 18:19:52 ubuntu-sata-51 kernel: [51267.628108] pci-stub 0000:03:12.4: claimed by stub syslog.ubuntu-sata-51 146910 Mar 25 18:19:52 ubuntu-sata-51 kernel: [51267.887635] pci-stub 0000:03:11.6: claimed by stub syslog.ubuntu-sata-51 146916 Mar 25 18:19:54 ubuntu-sata-51 kernel: [51270.133238] pci-stub 0000:03:13.1: enabling device (0000 -> 0002) syslog.ubuntu-sata-51 146918 Mar 25 18:19:56 ubuntu-sata-51 kernel: [51271.848116] pci-stub 0000:03:12.3: enabling device (0000 -> 0002) syslog.ubuntu-sata-51 146920 Mar 25 18:19:56 ubuntu-sata-51 kernel: [51271.950604] pci-stub 0000:03:12.4: enabling device (0000 -> 0002) syslog.ubuntu-sata-51 146922 Mar 25 18:19:56 ubuntu-sata-51 kernel: [51272.060462] pci-stub 0000:03:11.6: enabling device (0000 -> 0002) syslog.ubuntu-sata-51 146924 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.051439] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X syslog.ubuntu-sata-51 146925 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.051451] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X syslog.ubuntu-sata-51 146926 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.051458] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X syslog.ubuntu-sata-51 146927 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.260257] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X syslog.ubuntu-sata-51 146928 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.260272] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X syslog.ubuntu-sata-51 146929 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.260283] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X syslog.ubuntu-sata-51 146931 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.470006] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X syslog.ubuntu-sata-51 146932 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.470020] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X syslog.ubuntu-sata-51 146933 Mar 25 18:20:03 ubuntu-sata-51 kernel: [51279.470031] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X syslog.ubuntu-sata-51 146934 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.709613] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X syslog.ubuntu-sata-51 146935 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.709626] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X syslog.ubuntu-sata-51 146936 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.709638] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X syslog.ubuntu-sata-51 146937 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.830499] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X syslog.ubuntu-sata-51 146938 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.830512] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X syslog.ubuntu-sata-51 146939 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.830527] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X syslog.ubuntu-sata-51 146940 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.979145] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X syslog.ubuntu-sata-51 146941 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.979158] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X syslog.ubuntu-sata-51 146942 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51279.979170] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X syslog.ubuntu-sata-51 146943 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51280.069021] pci-stub 0000:03:11.6: irq 192 for MSI/MSI-X syslog.ubuntu-sata-51 146944 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51280.069035] pci-stub 0000:03:11.6: irq 193 for MSI/MSI-X syslog.ubuntu-sata-51 146945 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51280.069048] pci-stub 0000:03:11.6: irq 194 for MSI/MSI-X syslog.ubuntu-sata-51 146946 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51280.208841] pci-stub 0000:03:11.6: irq 192 for MSI/MSI-X syslog.ubuntu-sata-51 146947 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51280.208854] pci-stub 0000:03:11.6: irq 193 for MSI/MSI-X syslog.ubuntu-sata-51 146948 Mar 25 18:20:04 ubuntu-sata-51 kernel: [51280.208866] pci-stub 0000:03:11.6: irq 194 for MSI/MSI-X After the instance starts, it modifies udev rules for consistent interface naming and reloads the ixgbevf driver, in order for the new udev rules to apply. In some cases this leads to a situation, in which one of the VFs ends up with no IRQ assigned: syslog.ubuntu-sata-51 146952 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.618421] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X syslog.ubuntu-sata-51 146953 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.618436] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X syslog.ubuntu-sata-51 146954 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.618448] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X syslog.ubuntu-sata-51 146955 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.818060] pci-stub 0000:03:13.1: irq 99 for MSI/MSI-X syslog.ubuntu-sata-51 146956 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.818074] pci-stub 0000:03:13.1: irq 100 for MSI/MSI-X syslog.ubuntu-sata-51 146957 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.818086] pci-stub 0000:03:13.1: irq 101 for MSI/MSI-X syslog.ubuntu-sata-51 146958 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.957754] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X syslog.ubuntu-sata-51 146959 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.957768] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X syslog.ubuntu-sata-51 146960 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51293.957779] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X syslog.ubuntu-sata-51 146961 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.237367] pci-stub 0000:03:12.3: irq 180 for MSI/MSI-X syslog.ubuntu-sata-51 146962 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.237380] pci-stub 0000:03:12.3: irq 181 for MSI/MSI-X syslog.ubuntu-sata-51 146963 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.237391] pci-stub 0000:03:12.3: irq 182 for MSI/MSI-X syslog.ubuntu-sata-51 146964 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.309212] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X syslog.ubuntu-sata-51 146965 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.309227] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X syslog.ubuntu-sata-51 146966 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.309242] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X syslog.ubuntu-sata-51 146967 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.516813] pci-stub 0000:03:12.4: irq 183 for MSI/MSI-X syslog.ubuntu-sata-51 146968 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.516826] pci-stub 0000:03:12.4: irq 184 for MSI/MSI-X syslog.ubuntu-sata-51 146969 Mar 25 18:20:18 ubuntu-sata-51 kernel: [51294.516838] pci-stub 0000:03:12.4: irq 185 for MSI/MSI-X The VF at 03:11.6 at this point has no IRQ assigned, and is not functional at all within the instance. At this point, I don't know how to further debug this issue. In one case after instance reloaded ixgbevf, I saw that same IRQ was assigned to two different VFs - see IRQ 144/5/6: syslog.ccmaster 75318 Mar 22 17:25:49 ccmaster kernel: [12300.850946] pci-stub 0000:03:12.7: irq 81 for MSI/MSI-X syslog.ccmaster 75319 Mar 22 17:25:49 ccmaster kernel: [12300.850966] pci-stub 0000:03:12.7: irq 82 for MSI/MSI-X syslog.ccmaster 75320 Mar 22 17:25:49 ccmaster kernel: [12300.850982] pci-stub 0000:03:12.7: irq 83 for MSI/MSI-X syslog.ccmaster 75321 Mar 22 17:25:50 ccmaster kernel: [12301.060561] pci-stub 0000:03:12.7: irq 81 for MSI/MSI-X syslog.ccmaster 75322 Mar 22 17:25:50 ccmaster kernel: [12301.060576] pci-stub 0000:03:12.7: irq 82 for MSI/MSI-X syslog.ccmaster 75323 Mar 22 17:25:50 ccmaster kernel: [12301.060586] pci-stub 0000:03:12.7: irq 83 for MSI/MSI-X syslog.ccmaster 75324 Mar 22 17:25:50 ccmaster kernel: [12301.150203] pci-stub 0000:03:14.2: irq 120 for MSI/MSI-X syslog.ccmaster 75325 Mar 22 17:25:50 ccmaster kernel: [12301.150220] pci-stub 0000:03:14.2: irq 121 for MSI/MSI-X syslog.ccmaster 75326 Mar 22 17:25:50 ccmaster kernel: [12301.150235] pci-stub 0000:03:14.2: irq 122 for MSI/MSI-X syslog.ccmaster 75327 Mar 22 17:25:50 ccmaster kernel: [12301.339964] pci-stub 0000:03:14.2: irq 120 for MSI/MSI-X syslog.ccmaster 75328 Mar 22 17:25:50 ccmaster kernel: [12301.339977] pci-stub 0000:03:14.2: irq 121 for MSI/MSI-X syslog.ccmaster 75329 Mar 22 17:25:50 ccmaster kernel: [12301.339988] pci-stub 0000:03:14.2: irq 122 for MSI/MSI-X syslog.ccmaster 75330 Mar 22 17:25:50 ccmaster kernel: [12301.509569] pci-stub 0000:03:11.4: irq 144 for MSI/MSI-X syslog.ccmaster 75331 Mar 22 17:25:50 ccmaster kernel: [12301.509582] pci-stub 0000:03:11.4: irq 145 for MSI/MSI-X syslog.ccmaster 75332 Mar 22 17:25:50 ccmaster kernel: [12301.509593] pci-stub 0000:03:11.4: irq 146 for MSI/MSI-X syslog.ccmaster 75333 Mar 22 17:25:50 ccmaster kernel: [12301.799293] pci-stub 0000:03:10.1: irq 144 for MSI/MSI-X syslog.ccmaster 75334 Mar 22 17:25:50 ccmaster kernel: [12301.799309] pci-stub 0000:03:10.1: irq 145 for MSI/MSI-X syslog.ccmaster 75335 Mar 22 17:25:50 ccmaster kernel: [12301.799324] pci-stub 0000:03:10.1: irq 146 for MSI/MSI-X syslog.ccmaster 75336 Mar 22 17:25:51 ccmaster kernel: [12301.948971] pci-stub 0000:03:10.1: irq 144 for MSI/MSI-X syslog.ccmaster 75337 Mar 22 17:25:51 ccmaster kernel: [12301.948984] pci-stub 0000:03:10.1: irq 145 for MSI/MSI-X syslog.ccmaster 75338 Mar 22 17:25:51 ccmaster kernel: [12301.948996] pci-stub 0000:03:10.1: irq 146 for MSI/MSI-X At this point, the 03:11.4 VF became non-functional within the instance, and it has no IRQ assigned on the physical machine. Can anybody please advise how can I debug this issue further? Thanks! P.S.: I am not subscribed to the list, so pls reply directly -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
