On 10/05/2016 11:50 AM, Richard W.M. Jones wrote:
I was asked to look at the impact on boot times of adding (empty) PCIe root ports to the device model. The proposal from Laine is to add a few of these to every guest to allow hotplugging. Last time I looked into this I found that probing any (legacy) PCI device is expensive because of the inefficient way that qemu emulates accesses to PCI config space, requiring IIRC 2 or 4 VMEXITs to access every word. (PCI slots which are not occupied are basically free, the problem is PCI devices). At that time I did not look at Q35/PCIe at all. We generally aim for boot times under 600-700ms. Probing PCI devices takes a significant fraction of this time. The detailed analysis is attached. It comes from a program called 'boot-analysis' (https://github.com/libguestfs/libguestfs/tree/master/utils/boot-analysis). It is best viewed using 'less -r' so that you can see the colours. The summary table is: Number of ports 0 1 2 3 4 bios:overhead 874 884 875 878 935 kernel:entry 159 163 165 174 491 kernel:initcalls-before-userspace 1065 1090 1110 1147 1263 /init:udev-overhead 173 187 185 193 301 insmod virtio_pci 43 41 41 41 74 TOTAL + 51 + 62 + 119 + 750 (All times are in milliseconds.) A few observations: (1) For #ports <= 3, each extra port adds around 30-40ms to the boot time, which is consistent with what I saw when I measured legacy PCI.
Thanks for doing this (and especially on such short notice!). This is interesting info. I'm wondering now if the discontinuity is due to going over a threshold in the number of pcie-root-ports, the number of total PCI controllers, or the total number of devices. This is significant because a couple of the patches in the "Use more PCIe less legacy PCI" patchset I sent a two weeks ago will eliminate both the dmi-to-pci-bridge and the pci-bridge in the default config (as long as there are no legacy PCI devices). If the "jump" is due to an increase in total devices, or total PCI controllers, then getting rid of those two will change the number of root-ports that can be added prior to the jump.
Even if this is the case, it seems like we still want to have as little fat as possible.
(2) There is a sudden, unexplained and reproducible discontinuity when going from 3 to 4 ports. (Because libguestfs uses other devices, this might not actually be 3 to 4 PCIe devices, this is spare ports after all the others added for libguestfs devices.) (3) "kernel:entry" covers a lot of initialization that happens before the kernel prints its version string. This is moderately affected by adding PCIe ports until we hit the discontinuity. Based on this analysis I conclude: (a) Unless we can explain the discontinuity, avoid adding more than 3 root ports. (b) It would be nice to turn the whole thing off for people who don't care about / need hotplugging.
I had contemplated having an "availablePCIeSlots" (or something like that) that was either an attribute of the config, or an option in qemu.conf or libvirtd.conf. If we had such a setting, it could be set to "0".
(c) We could make PCI probing a lot faster by having the kernel handle PCI config space.
-- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list