On 24/04/15 17:18, Will Deacon wrote: > On Wed, Apr 22, 2015 at 08:41:02PM +0100, Stuart Yoder wrote: >>>> However, there is an improvement we envision as possible due to >>>> the limited number of SMMU contexts (i.e. 64). If there are >>>> 64 SMMU context registers it means that there is a max of >>>> 64 software contexts where things can be isolated. But, say I have >>>> an SRIOV card with 64 VFs, and I want to assign 8 of the VFs >>>> to a KVM VM. Those 8 PCI devices could share the same >>>> streamID/ITS-device-ID since they all share the same isolation >>>> context. >>>> >>>> What would be nice is at the time the 8 VFS are being added >>>> to the IOMMU domain is for the pcidevid -> streamID mapping >>>> table to be updated dynamically. It simply lets us make >>>> more efficient use of the limited streamIDs we have. >>>> >>>> I think it is this improvement that Minghuan had in mind >>>> in this patch. >>> >>> Ok, but in this case it should be possible to use a single context bank for >>> all of the VF streamIDs by configuring the appropriate SMR, no? >> >> Yes, but there are limited SMRs. In our case there are only >> 128 SMRs in SMMU-500 and we have potentially way more masters than >> that. > > Right, but you still only have 64 context banks at the end of the day, so do > you really anticipate having more than 128 masters concurrently using the > SMMU? If so, then we have devices sharing context banks so we could consider > reusing SMRs across masters, but historically that's not been something that > we've managed to solve. > >>> Wouldn't >>> that sort of thing be preferable to dynamic StreamID assignment? It would >>> certainly make life easier for the MSIs. >> >> It would be preferable, but given only 128 total stream IDS and >> 64 context registers it's potentially an issue. On our LS2085 SoC it is >> PCI and the fsl-mc bus (see description here: >> https://lkml.org/lkml/2015/3/5/795) that potentially have way >> more masters than streamIDS. So, for those busses we would essentially >> view a streamID as a "context ID"-- each SMR is associated with >> 1 context bank register. >> >> For PCI we have a programmable "PCI req ID"-to-"stream ID" >> mapping table in the PCI controller that is dynamically >> programmable. >> >> Looking at it like that means that we could have >> any number of masters but only 64 "contexts" >> and since the masters all all programmable it's >> seems feasbile to envision doing some bus/vendor >> specific set up when a device is added to an >> IOMMU domain. One thing that would need to be conveyed >> to the SMMU driver if doing dynamic streamID setup >> is what streamIDs are available to be used. > > Ok, but this is going to make life difficult for the MSI people, I suspect. > > Marc...? We're really facing two conflicting requirements: in order to minimize SMR usage, we want to alias multiple ReqIDs to a single StreamID, but in order to efficiently deal with MSIs, we want to see discrete DeviceIDs (the actual ReqIDs). I don't easily see how we reconcile the two. We can deal with the aliasing, provided that we extend the level of quirkiness that pci_for_each_dma_alias can deal with. But that doesn't solve any form of hotplug/SR-IOV behaviour. Somehow, we're going to end-up with grossly oversized ITTs, just to accommodate for the fact that we have no idea how many MSIs we're going to end-up needing. I'm not thrilled with that prospect. Thanks, M. -- Jazz is not dead. It just smells funny... -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
