> From: Jacob Pan <jacob.jun.pan@xxxxxxxxxxxxxxx> > Sent: Friday, March 10, 2023 1:06 AM > > Hi Jason, > > On Mon, 6 Mar 2023 15:05:27 -0400, Jason Gunthorpe <jgg@xxxxxxxxxx> > wrote: > > > On Mon, Mar 06, 2023 at 06:48:43PM +0000, Luck, Tony wrote: > > > >> ENQCMDS does not have the restriction of using a single CPU MSR to > > > >> store PASIDs, PASID is supplied to the instruction operand. > > > > > > > > Huh? That isn't what it says in the programming manual. It says the > > > > PASID only comes from the IA32_PASID msr and the only two operands > are > > > > the destination MMIO and the memory source for the rest of the > > > > payload. > > > > > > Jason, > > > > > > Two different instructions with only one letter different in the name. > > > > > > ENQCMD - ring 3 instruction. The PASID is inserted into the descriptor > > > pushed to the device from the IA32_PASID MSR. > > > > > > ENQCMDS - ring 0 instruction (see that trailing "S" for Supervisor > > > mode). In this case the submitter can include any PASID value they want > > > in the in-memory copy of the descriptor and ENQCMDS will pass that to > > > the device. > > > > Ah, well, my comment wasn't talking about ENQCMDS :) > > > > If ENQCMDS can take in an arbitary PASID then there is no > > justification here to use the global allocator. > > > > The rational is more like: > > > > IDXD uses PASIDs that come from the SVA allocator. It needs to create > > an internal kernel-only PASID that is non-overlapping so allow the SVA > > allocator to reserve PASIDs for driver use. > > > > IDXD has to use the global SVA PASID allocator beacuse its userspace > > will use ENQCMD which requires global PASIDs. > > > yes, great summary. I think that is the same as what I was trying to say > earlier :) > "due the unforgiving nature of ENQCMD that requires global PASIDs, > ENQCMDS > has no choice but to allocate from the same numberspace to avoid conflict." > > In that sense, I feel the global allocator should be staying with SVA > instead of moving to iommu core (as Kevin suggested). Because we are trying > to have non-overlapping pasid with SVA. > I still doubt 'reserve' is the right interface to define. for DMA domain probably yes as it's static and one-off. but thinking louder when the same driver starts to support SIOV we need allocating additional PASIDs on demand which is hardly to be fit in a reservation interface.
