> From: Jason Gunthorpe <jgg@xxxxxxxxxx> > Sent: Saturday, May 13, 2023 5:02 AM > > On Fri, May 12, 2023 at 02:59:40AM +0000, Tian, Kevin wrote: > > > From: Baolu Lu <baolu.lu@xxxxxxxxxxxxxxx> > > > Sent: Thursday, May 11, 2023 7:34 PM > > > > > > On 5/11/23 3:27 PM, Tian, Kevin wrote: > > > >> From: Alex Williamson<alex.williamson@xxxxxxxxxx> > > > >> Sent: Thursday, May 11, 2023 1:25 AM > > > >> > > > >> On Tue, 9 May 2023 08:34:53 +0000 > > > >> "Tian, Kevin"<kevin.tian@xxxxxxxxx> wrote: > > > >> > > > >>> According to PCIe spec (7.8.9 PASID Extended Capability Structure): > > > >>> > > > >>> The PASID configuration of the single non-VF Function representing > > > >>> the device is also used by all VFs in the device. A PF is permitted > > > >>> to implement the PASID capability, but VFs must not implement it. > > > >>> > > > >>> To enable PASID on VF then one open is where to locate the PASID > > > >>> capability in VF's vconfig space. vfio-pci doesn't know which offset > > > >>> may contain VF specific config registers. Finding such offset must > > > >>> come from a device specific knowledge. > > > >> Backup for a moment, VFs are governed by the PASID capability on the > > > >> PF. The PASID capability exposes control registers that imply the > > > >> ability to manage various feature enable bits. The VF owner does not > > > >> have privileges to manipulate those bits. For example, the PASID > Enable > > > >> bit should restrict the endpoint from sending TLPs with a PASID prefix, > > > >> but this can only be changed at the PF level for all associated VFs. > > > >> > > > >> The protocol specified in 7.8.9.3 defines this enable bit as RW. How do > > > >> we virtualize that? Either it's virtualized to be read-only and we > > > >> violate the spec or we allow it to be read-write and it has no effect, > > > >> which violates the spec. > > > >> > > > > Currently the PASID cap is enabled by default when a device is probed > > > > by iommu driver. Leaving it enabled in PF while guest wants it disabled > > > > in VF is harmless. W/o proper setup in iommu side the VF cannot > > > > do real work with PASID. > > > > > > [sorry for partial reply] > > > > > > I am attempting to move PASID enabling/disabling out of the iommu > > > driver and give its control to the device driver. One puzzle thing is > > > that PCI spec requires PASID control bits not changed once the ATS is > > > is enabled. So I also need to add iommu interfaces to enable/disable > > > ATS on devices. > > > > > > By default, the ATS is enabled by the iommu subsystem to utilize the > > > device TLB, the device driver needs to disable it before change the > > > PASID control bits and re-enable it afterwards. > > > > > > > ATS is also relied on by PRS. Not sure how that will be affected when > > ATS is dynamically turned on/off. and PRS is not tied to PASID. > > > > Jason, is it still a strong requirement to have driver-opt model for > > pasid enabling? iirc it's first raised in a discussion for an AMD GPU > > scenario [1] > > It is sounding worse and worse as we go along.. > > AMD and ARM both have the issue that the iommu settings and domain > types depend on if the driver intends to use PASID or not. There is > some small performance win if PASID is not used and the iommu driver > knows it is not used. > > We also get into some trouble with groups, I think, where it will be > hard for the iommu driver to know which mode to use when creating a > domain.. > > But, if the PASID control for a VF is on the PF then I think it is > hopeless. The iommu or PCI layers need to make these decisions and > drivers have to live with it. No PASID support unless the iommu turned > it on. > > This still suggests there would be some driver call to the iommu side > to check that PASID is setup. > yes, that still makes sense.
