On 11/4/22 13:03, Peter Krempa wrote:
On Fri, Nov 04, 2022 at 16:43:00 +0600, Oleg Vasilev wrote:
Hotplugging PCI devices on pc-i440fx machines is supported without
additional configuration. On q35, pcie-to-pci-bridge needs to be added
prior to the machine startup in order to support hotplug [1].
The idea is to support the same workflow for creating VMs in q35, as was
in pc. Namely, do not require additional configuration when hotplugging
is needed. Otherwise, all libvirt clients need to be updated which there
are a lot and they are maintained by different parties.
Instead, a pcie-to-pci-bridge better be created by default, so that PCI
slots are readily available. Might be a good idea to make it configurable
in the future.
Previously there was a pci-bridge present by default, but was removed in
d5fb8f4564 (qemu: don't add pci-bridge to Q35/arm domains unless it's needed, 2016-04-22)
[1]: https://libvirt.org/pci-hotplug.html
Signed-off-by: Oleg Vasilev <oleg.vasilev@xxxxxxxxxxxxx>
---
[...]
73 files changed, 621 insertions(+), 361 deletions(-)
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index 9ef6c8bb64..52a393e07f 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -3890,6 +3890,7 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
bool addImplicitSATA = false;
bool addPCIRoot = false;
bool addPCIeRoot = false;
+ bool addPCIeToPCIBridge = false;
bool addDefaultMemballoon = true;
bool addDefaultUSBKBD = false;
bool addDefaultUSBMouse = false;
@@ -3910,6 +3911,8 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
if (qemuDomainIsQ35(def)) {
addPCIeRoot = true;
addImplicitSATA = true;
+ if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_PCIE_PCI_BRIDGE))
+ addPCIeToPCIBridge = true;
/* Prefer adding a USB3 controller if supported, fall back
* to USB2 if there is no USB3 available, and if that's
@@ -4046,11 +4049,7 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
}
}
- /* When a machine has a pcie-root, make sure that there is always
- * a dmi-to-pci-bridge controller added as bus 1, and a pci-bridge
- * as bus 2, so that standard PCI devices can be connected
- *
- * NB: any machine that sets addPCIeRoot to true must also return
+ /* NB: any machine that sets addPCIeRoot to true must also return
* true from the function qemuDomainSupportsPCI().
*/
if (addPCIeRoot) {
@@ -4069,6 +4068,16 @@ qemuDomainDefAddDefaultDevices(virQEMUDriver *driver,
}
}
+ /* Add a pcie-to-pci-bridge and pcie-root-port to plug it into. */
+ if (addPCIeToPCIBridge) {
+ if (virDomainDefMaybeAddController(def, VIR_DOMAIN_CONTROLLER_TYPE_PCI, 1,
+ VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT_PORT) < 0)
+ return -1;
+ if (virDomainDefMaybeAddController(def, VIR_DOMAIN_CONTROLLER_TYPE_PCI, 2,
+ VIR_DOMAIN_CONTROLLER_MODEL_PCIE_TO_PCI_BRIDGE) < 0)
+ return -1;
+ }
Doing this unconditionally breaks ABI of the VM, as new guest visible
devices appear without any change in configuration. What's worse this
implementation would also break migration compatibility from older
versions of libvirt as there isn't any form of explicit handling for it.
Hmm, I am not so sure here. Doing this we goes exactly in the same way
as done for balloon and thus my expectations are that the device will
appear in the configuration.
Quick check over domain.xml seems that my opinion is correct. I am
able to find the following device
<controller type='pci' index='2' model='pcie-to-pci-bridge'>
<model name='pcie-pci-bridge'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</controller>
I unfortunately don't see any reasonable solution originating in
libvirt. The bridge will need to be added by users or higher level
management tools (e.g. virt-install) as libvirt explicitly wants to
support use cases when VMs are started without persisting the XML
definition (e.g. always re-define the XML before start or use the
'createXML' api).
That approach seems cumbersome. Even in Virtuozzo we have
* QA test client, which creates VMs on its own
* 'virt-install' based software
* 'prl-disp-service' based software
* OpenStack based clusters
With your proposal we would need to implement a code,
which will check that the VM is created with Q35
machine type and in that case the bridge is added
each time the virtual machine is created. There are a lot
of software doing that and the process seems to complex.
In general, old PC based software have supported ordinary
VirtIO network device hotplug without any VM specific
pre-configuration and that was brilliant. Please correct
me if I am wrong.
Proposed approach requires a lot of efforts and seems
error prone.
On the other hand I could propose at least 2 options:
* hotplugging of PCIe-to-PCI bridge at the time
of the first hotlug to Q35 VM
* put the behavior of creating default bridge under
the option in libvirt configuration
In general, hotplugging of bridge seems much more
guest specific but potentially could work.
Den
