On 12/9/2023 8:47 AM, Jason Gunthorpe wrote:
On Fri, Nov 17, 2023 at 05:07:11AM -0800, Yi Liu wrote:
Take Intel VT-d as an example, the stage-1 translation table is I/O page
table. As the below diagram shows, guest I/O page table pointer in GPA
(guest physical address) is passed to host and be used to perform the stage-1
address translation. Along with it, modifications to present mappings in the
guest I/O page table should be followed with an IOTLB invalidation.
I've been looking at what the three HW's need for invalidation, it is
a bit messy.. Here is my thinking. Please let me know if I got it right
What is the starting point of the guest memory walks:
Intel: Single Scalable Mode PASID table entry indexed by a RID & PASID
AMD: GCR3 table (a table of PASIDs) indexed by RID
GCR3 table is indexed by PASID.
Device Table (DTE) is indexted by DeviceID (RID)
...
Will ATC be forwarded or synthesized:
Intel: The (vDomain-ID,PASID) is a unique nesting domain so
the hypervisor knows exactly which RIDs this nesting domain is
linked to and can generate an ATC invalidation. Plan is to
supress/discard the ATC invalidations from the VM and generate
them in the hypervisor.
AMD: (vDomain-ID,PASID) is ambiguous, it can refer to multiple GCR3
tables. We know which maximal set of RIDs it represents, but not
the actual set. I expect AMD will forward the ATC invalidation
to avoid over invalidation.
Not sure I understand your description here.
For the AMD IOMMU INVALIDE_IOMMU_PAGES (i.e. invalidate the IOMMU TLB),
the hypervisor needs to map gDomainId->hDomainId and issue the command
on behalf of the VM along with the PASID and GVA (or GVA range) provided
by the guest.
For the AMD IOMMU INVALIDE_IOTLB_PAGES (i.e. invalidate the ATC on the
device), the hypervisor needs to map gDeviceId->hDeviceId and issue the
command on behalf of the VM along with the PASID and GVA (or GVA range)
provided by the guest.
ARM: ASID is ambiguous. We have no idea which Nesting Domain/CD table
the ASID is contained in. ARM must forward the ATC invalidation
from the guest.
What iommufd object should receive the IOTLB invalidation command list:
Intel: The Nesting domain. The command list has to be broken up per
(vDomain-ID,PASID) and that batch delivered to the single
nesting domain. Kernel ignores vDomain-ID/PASID and just
invalidates whatever the nesting domain is actually attached to
AMD: Any Nesting Domain in the vDomain-ID group. The command list has
to be broken up per (vDomain-ID). Kernel replaces
vDomain-ID with pDomain-ID from the nesting domain and executes
the invalidation.
ARM: The Nesting Parent domain. Kernel forces the VMID from the
Nesting Parent and executes the invalidation.
In all cases the VM issues an ATC invalidation with (vRID, PASID) as
the tag. The VMM must translate vRID -> dev_id -> pRID
For a pure SW flow the vRID can be mapped to the dev_id and the ATC
invalidation delivered to the device object (eg IOMMUFD_DEV_INVALIDATE)
Finally, we have the HW driven invalidation DMA queues that can be
directly assigned to the guest. AMD and SMMUv3+vCMDQ support this. In
this case the HW is directly processing invalidation commands without
a hypervisor trap.
To make this work the iommu needs to be programmed with:
AMD: A vDomain-ID -> pDomain-ID table
A vRID -> pRID table
This is all bound to some "virtual function"
By "virtual function", I assume you are referring to the AMD vIOMMU
instance in the guest?
ARM: A vRID -> pRID table
The vCMDQ is bound to a VM_ID, so to the Nesting Parent
For AMD, as above, I suggest the vDomain-ID be passed when creating
the nesting domain
Sure, we can do this part.
The AMD "virtual function".. It is probably best to create a new iommufd
object for this and it can be passed in to a few places
Something like IOMMUFD_OBJ_VIOMMU? Then operation would include
something like:
* Init
* Destroy
* ...
The vRID->pRID table should be some mostly common
IOMMUFD_DEV_ASSIGN_VIRTUAL_ID. AMD will need to pass in the virtual
function ID and ARM will need to pass in the Nesting Parent ID.
Ok.
...
Thus next steps:
- Respin this and lets focus on Intel only (this will be tough for
the holidays, but if it is available I will try)
- Get an ARM patch that just does IOTLB invalidation and add it to my
part 3
- Start working on IOMMUFD_DEV_INVALIDATE along with an ARM
implementation of it
- Reorganize the AMD RFC broadly along these lines and lets see it
freshened up in the next months as well. I would like to see the
AMD support structured to implement the SW paths in first steps and
later add in the "virtual function" acceleration stuff. The latter
is going to be complex.
Working on refining the part 1 to add HW info reporting and nested
translation (minus the invalidation stuff). Should be sending out soon.
Suravee
