> From: Tian, Kevin > Sent: Friday, May 12, 2023 10:53 AM > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx> > > Sent: Thursday, May 11, 2023 11:45 PM > > > > On Thu, 11 May 2023 07:27:27 +0000 > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote: > > > > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx> > > > > Sent: Thursday, May 11, 2023 1:25 AM > > > > > > > > On Tue, 9 May 2023 08:34:53 +0000 > > > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote: > > > > > > > > > According to PCIe spec (7.8.9 PASID Extended Capability Structure): > > > > > > > > > > The PASID configuration of the single non-VF Function representing > > > > > the device is also used by all VFs in the device. A PF is permitted > > > > > to implement the PASID capability, but VFs must not implement it. > > > > > > > > > > To enable PASID on VF then one open is where to locate the PASID > > > > > capability in VF's vconfig space. vfio-pci doesn't know which offset > > > > > may contain VF specific config registers. Finding such offset must > > > > > come from a device specific knowledge. > > > > > > > > Backup for a moment, VFs are governed by the PASID capability on the > > > > PF. The PASID capability exposes control registers that imply the > > > > ability to manage various feature enable bits. The VF owner does not > > > > have privileges to manipulate those bits. For example, the PASID > Enable > > > > bit should restrict the endpoint from sending TLPs with a PASID prefix, > > > > but this can only be changed at the PF level for all associated VFs. > > > > > > > > The protocol specified in 7.8.9.3 defines this enable bit as RW. How do > > > > we virtualize that? Either it's virtualized to be read-only and we > > > > violate the spec or we allow it to be read-write and it has no effect, > > > > which violates the spec. > > > > > > > > > > Currently the PASID cap is enabled by default when a device is probed > > > by iommu driver. Leaving it enabled in PF while guest wants it disabled > > > in VF is harmless. W/o proper setup in iommu side the VF cannot > > > do real work with PASID. > > > > > > From this angle fully virtualizing it in software looks good to me. > > > > So you're suggesting that the IOMMU setup for the VF to make use of > > PASID would not occur until or unless PASID Enable is set in the > > virtualized VF PASID capability and that support would be torn down > > when PASID Enable is cleared? > > No that is not the case. The IOMMU setup is initiated by vIOMMU > and orthogonal to the PASID cap virtualization. > > Following the current IOMMU behavior as Baolu described the guest > will always enable vPASID in the vIOMMU driver. > > Even if the guest implements an driver-opt model for vPASID enabling, > in typical case the guest driver will not request vIOMMU setup for VF > PASIDs if it doesn't intend to enable vPASID cap. In this case the > physical IOMMU is left blocking VF PASIDs hence leaving PF PASID enabled > doesn't hurt. > > If an insane guest driver does try to enable vIOMMU PASID (so VF PASIDs > are allowed in physical IOMMU) while leaving vPASID disabled in VF, > I'm not sure what would be the actual problem leaving PF PASID enabled. > The guest driver kind of wants to fool itself by already setting up the > fabric to allow VF PASID but then block PASID in VF itself? > > > > > This is still not strictly in adherence with the definition of the > > PASID Enable bit which specifies that this bit controls whether the > > endpoint is able to send or receive TLPs with the PASID prefix, which > > clearly virtualization interacting with the IOMMU to block or allow > > PASIDs from the VF RID cannot change. Is it sufficient? > > > > For example we can't use the vPASID capability to make any guarantees > > about in-flight PASID TLPs when sequencing IOMMU operations since we > > can't actually prevent VFs using PASID so long as PASID Enable is set > > on the PF. > > IOMMU cares about in-flight PASID TLPs only when it's unblocked. > > If it's already blocked then it doesn't matter whether VF is sending PASID > TLP or not. > > btw think about the current situation. Even if vfio-pci doesn't expose > PASID cap today, it's physically enabled by iommu driver already. Then > the guest is already able to program the device to send PASID TLP's. > > fully virtualizing vPASID cap just aligns with this fact. 😊 > Hi, Alex, If you are OK with above explanation we can continue discussing how to expose the PASID cap for VF. At the start I listed several options to quirk the offset in the kernel. Jason suggested that the kernel should not expose the cap unconditionally. Then I proposed it could be done via a device feature and leave the offset to be quirked in VMM. [1] Is it a reasonable way to go? Thanks Kevin [1] https://lore.kernel.org/lkml/BN9PR11MB5276AE43183A3AA6AB806A398C749@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/
