On 2024/9/27 14:02, Nicolin Chen wrote:
On Fri, Sep 27, 2024 at 01:38:08PM +0800, Yi Liu wrote:
Does it mean each vIOMMU of VM can only have
one s2 HWPT?
Giving some examples here:
- If a VM has 1 vIOMMU, there will be 1 vIOMMU object in the
kernel holding one S2 HWPT.
- If a VM has 2 vIOMMUs, there will be 2 vIOMMU objects in the
kernel that can hold two different S2 HWPTs, or share one S2
HWPT (saving memory).
So if you have two devices assigned to a VM, then you may have two
vIOMMUs or one vIOMMU exposed to guest. This depends on whether the two
devices are behind the same physical IOMMU. If it's two vIOMMUs, the two
can share the s2 hwpt if their physical IOMMU is compatible. is it?
Yes.
To achieve the above, you need to know if the physical IOMMUs of the
assigned devices, hence be able to tell if physical IOMMUs are the
same and if they are compatible. How would userspace know such infos?
My draft implementation with QEMU does something like this:
- List all viommu-matched iommu nodes under /sys/class/iommu: LINKs
- Get PCI device's /sys/bus/pci/devices/0000:00:00.0/iommu: LINK0
- Compare the LINK0 against the LINKs
We so far don't have an ID for physical IOMMU instance, which can
be an alternative to return via the hw_info call, otherwise.
intel platform has a kind of ID for the physical IOMMUs.
ls /sys/class/iommu/
dmar0 dmar1 dmar10 dmar11 dmar12 dmar13 dmar14 dmar15 dmar16
dmar17 dmar18 dmar19 dmar2 dmar3 dmar4 dmar5 dmar6 dmar7 dmar8
dmar9 iommufd_selftest_iommu.0
QEMU then does the routing to assign PCI buses and IORT (or DT).
This part is suggested now to move to libvirt though. So, I think
at the end of the day, libvirt would run the sys check and assign
a device to the corresponding pci bus backed by the correct IOMMU.
and also give the correct viommu for the device.
This gives an example showing two devices behind iommu0 and third
device behind iommu1 are assigned to a VM:
-device pxb-pcie.id=pcie.viommu0,bus=pcie.0.... \ # bus for viommu0
-device pxb-pcie.id=pcie.viommu1,bus=pcie.0.... \ # bus for viommu1
-device pcie-root-port,id=pcie.viommu0p0,bus=pcie.viommu0... \
-device pcie-root-port,id=pcie.viommu0p1,bus=pcie.viommu0... \
-device pcie-root-port,id=pcie.viommu1p0,bus=pcie.viommu1... \
-device vfio-pci,bus=pcie.viommu0p0... \ # connect to bus for viommu0
-device vfio-pci,bus=pcie.viommu0p1... \ # connect to bus for viommu0
-device vfio-pci,bus=pcie.viommu1p0... # connect to bus for viommu1
is the viommu# an "-object" or just hints to describe the relationship
between device and viommu and build the IORT?
I'm considering how it would look like if the QEMU Intel vIOMMU is going
to use the viommu obj. Currently, we only support one virtual VT-d due to
some considerations like hot-plug. Per your conversation with Kevin, it
seems to be supported. So there is no strict connection between vIOMMU
and vIOMMU obj. But the vIOMMU obj can only be connected with one pIOMMU.
right?
https://lore.kernel.org/linux-iommu/ZvYJl1AQWXWX0BQL@Asurada-Nvidia/
For compatibility to share a stage-2 HWPT, basically we would do
a device attach to one of the stage-2 HWPT from the list that VMM
should keep. This attach has all the compatibility test, down to
the IOMMU driver. If it fails, just allocate a new stage-2 HWPT.
yeah. I think this was covered by Zhenzhong's QEMU series.
--
Regards,
Yi Liu
