Hi Will, > -----Original Message----- > From: Will Deacon [mailto:will.deacon@xxxxxxx] > Sent: Wednesday, April 22, 2015 10:37 PM > To: Yoder Stuart-B08248 > Cc: Sethi Varun-B16395; Lian Minghuan-B31939; linux-pci@xxxxxxxxxxxxxxx; > Arnd Bergmann; Hu Mingkai-B21284; Zang Roy-R61911; Bjorn Helgaas; Wood > Scott-B07421; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx > Subject: Re: [PATCH 1/2] irqchip/gicv3-its: Support share device ID > > Hi Stuart, > > First of, thanks for taking the time to explain this in more detail. > Comments inline. > > On Fri, Apr 17, 2015 at 03:19:08PM +0100, Stuart Yoder wrote: > > > On Wed, Apr 15, 2015 at 02:18:13PM +0100, Varun Sethi wrote: > > > > Yes, deviceid=stream id (i.e. ICID + other bits). I am not sure if > > > > TBU ID would also be forwarded as a part of stream id to GIC. My > > > > understanding is that TBU ID is forwarded (as a part of the stream > > > > ID) to the TCU in case of a TBU translation miss. In case of the > > > > LS2085 PCIe controller you would have to setup the PCIe device ID > > > > to stream ID translation table. We may have to restrict the number > > > > of entries based on the available number of contexts. > > > > > > Unfortunately, I'm having a really hard time parsing this thread > > > (some parts of it simply don't make sense; others use > > > non-architectural terms and overall I don't get a feeling for the problem). > > > > > > Please could you explain your system design step by step so that I > > > can understand (a) what you've built and (b) why the current design > > > of Linux is causing you problems? > > > > > > Sorry if I'm just being thick, but it's important that we get this right. > > > > I'll try to summarize some key points about the system... > > > > System is using a single SMMU-500 (1 TCU, 6 TBUs) and GICv3-ITS. > > There are PCI, fsl-mc, and platform devices that do DMA. Devices on > > the PCI and fsl-mc bus generate message interrupts. > > Ah cool, so you have multiple buses sharing a single SMMU? That's going to > necessitate some ID remapping in the device-tree. Perhaps you could > comment on Mark Rutland's proposal if it does/doesn't work for you: > > http://lists.infradead.org/pipermail/linux-arm-kernel/2015- > March/333199.html > > > > The flow a message interrupt would take is this: > > > > -------------- > > PCI device > > -------------- > > | > > | pcidevid + MSI msg > > | > > V > > -------------- > > PCI controller > > pcidevid -> > > streamID > > mapping > > -------------- > > | > > | streamID + MSI msg > > | > > V > > -------------- > > SMMU > > -------------- > > | > > | streamID + MSI msg > > | > > V > > -------------- > > CCN-504 (Dickens) > > -------------- > > | > > | streamID + MSI msg > > | > > V > > The streamID here as the same as the one coming out of the SMMU, right? > (just trying to out why you have the CCN-504 in the picture). > > > -------------- > > GICv3 ITS streamID == ITS deviceID > > -------------- > > > > So, the way things work (at least initially) is that each PCI device > > maps to a single streamID, and thus each device has a separate ITT in > > the ITS. So, things should be cool. > > > > However, there is an improvement we envision as possible due to the > > limited number of SMMU contexts (i.e. 64). If there are > > 64 SMMU context registers it means that there is a max of > > 64 software contexts where things can be isolated. But, say I have an > > SRIOV card with 64 VFs, and I want to assign 8 of the VFs to a KVM VM. > > Those 8 PCI devices could share the same streamID/ITS-device-ID since > > they all share the same isolation context. > > > > What would be nice is at the time the 8 VFS are being added to the > > IOMMU domain is for the pcidevid -> streamID mapping table to be > > updated dynamically. It simply lets us make more efficient use of the > > limited streamIDs we have. > > > > I think it is this improvement that Minghuan had in mind in this > > patch. > > Ok, but in this case it should be possible to use a single context bank for all of > the VF streamIDs by configuring the appropriate SMR, no? Wouldn't that sort > of thing be preferable to dynamic StreamID assignment? It would certainly > make life easier for the MSIs. > Yes, that would happen when the all the VFs are attached to the same domain. But it would be an issue if we have to attach devices to different domains. -Varun -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
