Gentle ping in case it's buried in your mailbox. 😊 we'd like to settle it down soon as the conclusion may even affect PF (e.g. if all agree that device feature is the right interface to go). > From: Tian, Kevin > Sent: Wednesday, May 17, 2023 1:22 PM > > > From: Tian, Kevin > > Sent: Friday, May 12, 2023 10:53 AM > > > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx> > > > Sent: Thursday, May 11, 2023 11:45 PM > > > > > > On Thu, 11 May 2023 07:27:27 +0000 > > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote: > > > > > > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx> > > > > > Sent: Thursday, May 11, 2023 1:25 AM > > > > > > > > > > On Tue, 9 May 2023 08:34:53 +0000 > > > > > "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote: > > > > > > > > > > > According to PCIe spec (7.8.9 PASID Extended Capability Structure): > > > > > > > > > > > > The PASID configuration of the single non-VF Function representing > > > > > > the device is also used by all VFs in the device. A PF is permitted > > > > > > to implement the PASID capability, but VFs must not implement it. > > > > > > > > > > > > To enable PASID on VF then one open is where to locate the PASID > > > > > > capability in VF's vconfig space. vfio-pci doesn't know which offset > > > > > > may contain VF specific config registers. Finding such offset must > > > > > > come from a device specific knowledge. > > > > > > > > > > Backup for a moment, VFs are governed by the PASID capability on > the > > > > > PF. The PASID capability exposes control registers that imply the > > > > > ability to manage various feature enable bits. The VF owner does not > > > > > have privileges to manipulate those bits. For example, the PASID > > Enable > > > > > bit should restrict the endpoint from sending TLPs with a PASID prefix, > > > > > but this can only be changed at the PF level for all associated VFs. > > > > > > > > > > The protocol specified in 7.8.9.3 defines this enable bit as RW. How > do > > > > > we virtualize that? Either it's virtualized to be read-only and we > > > > > violate the spec or we allow it to be read-write and it has no effect, > > > > > which violates the spec. > > > > > > > > > > > > > Currently the PASID cap is enabled by default when a device is probed > > > > by iommu driver. Leaving it enabled in PF while guest wants it disabled > > > > in VF is harmless. W/o proper setup in iommu side the VF cannot > > > > do real work with PASID. > > > > > > > > From this angle fully virtualizing it in software looks good to me. > > > > > > So you're suggesting that the IOMMU setup for the VF to make use of > > > PASID would not occur until or unless PASID Enable is set in the > > > virtualized VF PASID capability and that support would be torn down > > > when PASID Enable is cleared? > > > > No that is not the case. The IOMMU setup is initiated by vIOMMU > > and orthogonal to the PASID cap virtualization. > > > > Following the current IOMMU behavior as Baolu described the guest > > will always enable vPASID in the vIOMMU driver. > > > > Even if the guest implements an driver-opt model for vPASID enabling, > > in typical case the guest driver will not request vIOMMU setup for VF > > PASIDs if it doesn't intend to enable vPASID cap. In this case the > > physical IOMMU is left blocking VF PASIDs hence leaving PF PASID enabled > > doesn't hurt. > > > > If an insane guest driver does try to enable vIOMMU PASID (so VF PASIDs > > are allowed in physical IOMMU) while leaving vPASID disabled in VF, > > I'm not sure what would be the actual problem leaving PF PASID enabled. > > The guest driver kind of wants to fool itself by already setting up the > > fabric to allow VF PASID but then block PASID in VF itself? > > > > > > > > This is still not strictly in adherence with the definition of the > > > PASID Enable bit which specifies that this bit controls whether the > > > endpoint is able to send or receive TLPs with the PASID prefix, which > > > clearly virtualization interacting with the IOMMU to block or allow > > > PASIDs from the VF RID cannot change. Is it sufficient? > > > > > > For example we can't use the vPASID capability to make any guarantees > > > about in-flight PASID TLPs when sequencing IOMMU operations since we > > > can't actually prevent VFs using PASID so long as PASID Enable is set > > > on the PF. > > > > IOMMU cares about in-flight PASID TLPs only when it's unblocked. > > > > If it's already blocked then it doesn't matter whether VF is sending PASID > > TLP or not. > > > > btw think about the current situation. Even if vfio-pci doesn't expose > > PASID cap today, it's physically enabled by iommu driver already. Then > > the guest is already able to program the device to send PASID TLP's. > > > > fully virtualizing vPASID cap just aligns with this fact. 😊 > > > > Hi, Alex, > > If you are OK with above explanation we can continue discussing how > to expose the PASID cap for VF. > > At the start I listed several options to quirk the offset in the kernel. > > Jason suggested that the kernel should not expose the cap unconditionally. > > Then I proposed it could be done via a device feature and leave the > offset to be quirked in VMM. [1] Is it a reasonable way to go? > > Thanks > Kevin > > [1] > https://lore.kernel.org/lkml/BN9PR11MB5276AE43183A3AA6AB806A398C74 > 9@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/
