On Thu, May 23, 2024 at 05:44:40AM +0000, Tian, Kevin wrote: > > From: Nicolin Chen <nicolinc@xxxxxxxxxx> > > Sent: Monday, May 13, 2024 12:39 PM > > > > On Sun, May 12, 2024 at 11:46:54AM -0300, Jason Gunthorpe wrote: > > > On Fri, Apr 12, 2024 at 08:47:04PM -0700, Nicolin Chen wrote: > > > > Add a pair of ops to set and unet device's virtual ID that belongs to > > > > a viommu object. They will be used, in the following patch, by iommufd > > > > to support some HW-acceleration feature from the host level. > > > > > > > > For instance, every device behind an ARM SMMU has a Stream ID. The ID > > > > is used by ATC invalidation commands so SMMU HW can direct > > invalidation > > > > requests to the corresponding PCI device where the ID belongs to. In a > > > > virtualization use case, a passthroughed device in the VM will have a > > > > virtuail Stream ID, used by the ATC invalidation commands in the guest > > > > system. NVIDIA's CMDQV extension for SMMUv3 provides a v-interface > > to > > > > execute the guest-level ATC invalidation commands directly, yet needs > > > > the HW to be aware of its virtual Stream ID so it can replace with its > > > > physical Stream ID. > > > > > > I imagine using this as well for the ATC invalidation commands. It > > > would be very easy and simplifying if the command fixup just extracted > > > the vSID from the ATC invalidation and used an xarray to turn it into > > > a pSID and then pushed the resulting command. > > > > You mean the nested SMMU series right? Actually the set_dev_id > > ioctl was a part of that until we wanted to try DEV_INVALIDATE. > > > > So again, yes, it makes sense to me that we move viommu and the > > set_dev_id to the nested series, and then drop DEV_INVALIDATE. > > I'm right about to ask how the nesting series is going. Per earlier > discussion iirc the nesting series will go in before VCMDQ? Yes. It still should. Yet we ended up with adding VIOMMU to the nested SMMU series too. A shared S2 domain/hwpt isn't exclusive for VCMDQ use case but also for regular nesting on a multi-SMMU setup. So, VIOMMU turns out to be the best object that we have at this moment to hold individual VMIDs for different physical SMMUs sharing a single S2 domain. Its virtual device ID lookup feature can also allow us to forget about DEV_INVALIDATE ioctl for now. Jason listed all the tasks ahead in this thread too, using SMMU as an example: > So we have this stuff still open: > - Identity STE with PASID (part 2b) > - IOMMU_GET_HW_INFO (part 3) > - IOMMU_HWPT_ALLOC_NEST_PARENT (part 3) > - IOMMU_HWPT_DATA_ARM_SMMUV3 (part 3) > - IOMMU_HWPT_INVALIDATE_DATA_ARM_SMMUV3 > - VIOMMU_ALLOC, VIOMMU_ATTACH > - VIOMMU_INVALIDATE By this series nesting setup is done. We need Baolu's solution or VQUEUE for fault reporting after that. Thanks Nicolin
