Hi Daniel,
Top level libvirt device representation in XML is based on the device
*class*, not the specific device impl. Adding a <nestedSmmuv3> device
type XML element in libvirt is totally inappropriate. Any configuration
must be done beneath the <iommu> element.
Would keeping track of PXB <=> host SMMU nodes be better represented with a
<nestedSmmuv3> PXB attribute like below, when the "nestedSmmuv3" IOMMU model
is specified? This method would be simplest IMO because we could omit
keeping track of the nestedSmmuv3 bus number in the virDomainDef struct
since its association with a PXB controller would already be baked-in.
<devices>
...
<controller type='pci' index='1' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='254'/>
<nestedSmmuv3>smmu3.0x0000000012000000</nestedSmmuv3>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
</controller>
...
<iommu model='nestedSmmuv3'/>
</devices>
Or would it still be preferred to purely contain the host SMMU node names
and bus numbers within the <iommu> element? If so, the virDomainDef struct
is setup to only have a single virDomainIOMMUDef member, but we need to keep
track of multiple host SMMU node names and bus numbers. Would we setup
multiple virDomainIOMMUDef members? Or add "char** nestedSmmuv3" and "size_t
*nestedSmmuv3Bus" members to the virDomainIOMMUDef struct to keep track of
these multiple host SMMU node names and bus numbers?
Or we could modify the device info member of virDomainIOMMUDef to be a
variable-length array of device info structs instead of the "size_t
*nestedSmmuv3Bus" member. But I'm not convinced this would be the cleanest
approach, and the qemu command line doesn't specify a slot and function to
arm-smmuv3-nested devices - it just specifies bus number to keep track of
which PXB is associated with each SMMU node.
If we need a way to associate PXB to SMMU node, we have multiple possible
approaches, listed below in order of best to worst in my opinion:
1. Adding a <nestedSmmuv3> attribute for PXB controller.
2. Having a single virDomainIOMMUDef struct for virDomainDef. Adding
variable-length array members to virDomainIOMMUDef for multiple SMMU node
names and bus numbers.
3. Having a single virDomainIOMMUDef struct for virDomainDef. Adding a
variable-length array member to virDomainIOMMUDef for SMMU node names.
Changing the single virDomainDeviceInfo struct to a variable-length array of
virDomainDeviceInfo structs for multiple nested SMMU bus numbers.
4. Supporting multiple virDomainIOMMUDef structs for virDomainDef.
I would appreciate your thoughts on which method to go with.
Ignoring the specific QEMU impl, can you give a general outline of the
relationship between host SMMU(s), guest SMMU(s) and PCI controllers,
especially the M:N values of the relations.
Guest SMMU (vSMMU) allows stage 1 translation use cases for the VM and
VM-assigned devices. For the multiple vSMMU design, a 1:1 ratio between
host SMMUs and vSMMUs is implemented where commands issued to each vSMMU
are translated by the hypervisor and forwarded to the corresponding host
SMMU.
The 1:1 ratio between vSMMUs and guest PCI controllers that directly
plug into the PCIe root controller (i.e. PXBs) is for the specific QEMU
implementation to make it easier to establish the mapping of which
passthrough device to which SMMU in the ACPI I/O remapping table while
avoiding associating emulated devices to vSMMUs [0].
Also, if this is getting associated with the host SMMU in some way,
does this have implications for live migration compatibility ?
Live migration of devices associated with vIOMMU is currently blocked,
per [1] and [2] below. Joao is working on it here [3].
[0]
https://lore.kernel.org/qemu-devel/bc7a57311ac4976699789ceca329edfdfe823c2d.1719361174.git.nicolinc@xxxxxxxxxx/
[1]
https://github.com/qemu/qemu/commit/3c26c80a0a269ce7870d1475e756607e939226cd
[2]
https://lore.kernel.org/qemu-devel/3fa6f093ff9a4749bcd25d0dfa60b1d7@xxxxxxxxxx/
[3]
https://lore.kernel.org/all/20230622214845.3980-1-joao.m.martins@xxxxxxxxxx/
Thanks,
Nathan
