On Tue, Apr 23, 2024 at 11:47:50PM +0000, Tian, Kevin wrote: > > From: Jason Gunthorpe <jgg@xxxxxxxxxx> > > Sent: Tuesday, April 23, 2024 8:02 PM > > > > On Tue, Apr 23, 2024 at 07:43:27AM +0000, Tian, Kevin wrote: > > > I'm not sure how userspace can fully handle this w/o certain assistance > > > from the kernel. > > > > > > So I kind of agree that emulated PASID capability is probably the only > > > contract which the kernel should provide: > > > - mapped 1:1 at the physical location, or > > > - constructed at an offset according to DVSEC, or > > > - constructed at an offset according to a look-up table > > > > > > The VMM always scans the vfio pci config space to expose vPASID. > > > > > > Then the remaining open is what VMM could do when a VF supports > > > PASID but unfortunately it's not reported by vfio. W/o the capability > > > of inspecting the PASID state of PF, probably the only feasible option > > > is to maintain a look-up table in VMM itself and assumes the kernel > > > always enables the PASID cap on PF. > > > > I'm still not sure I like doing this in the kernel - we need to do the > > same sort of thing for ATS too, right? > > VF is allowed to implement ATS. > > PRI has the same problem as PASID. I'm surprised by this, I would have guessed ATS would be the device global one, PRI not being per-VF seems problematic??? How do you disable PRI generation to get a clean shutdown? > > It feels simpler if the indicates if PASID and ATS can be supported > > and userspace builds the capability blocks. > > this routes back to Alex's original question about using different > interfaces (a device feature vs. PCI PASID cap) for VF and PF. I'm not sure it is different interfaces.. The only reason to pass the PF's PASID cap is to give free space to the VMM. If we are saying that gaps are free space (excluding a list of bad devices) then we don't acutally need to do that anymore. VMM will always create a synthetic PASID cap and kernel will always suppress a real one. An iommufd query will indicate if the vIOMMU can support vPASID on that device. Same for all the troublesome non-physical caps. > > There are migration considerations too - the blocks need to be > > migrated over and end up in the same place as well.. > > Can you elaborate what is the problem with the kernel emulating > the PASID cap in this consideration? If the kernel changes the algorithm, say it wants to do PASID, PRI, something_new then it might change the layout We can't just have the kernel decide without also providing a way for userspace to say what the right layout actually is. :\ > Does it talk about a case where the devices between src/dest are > different versions (but backward compatible) with different unused > space layout and the kernel approach may pick up different offsets > while the VMM can guarantee the same offset? That is also a concern where the PCI cap layout may change a bit but they are still migration compatible, but my bigger worry is that the kernel just lays out the fake caps in a different way because the kernel changes. At least if the VMM is doing this then the VMM can include the information in its migration scheme and use it to recreate the PCI layout withotu having to create a bunch of uAPI to do so. Jason
