On Tue, 8 May 2018 13:45:50 -0600 Logan Gunthorpe <logang@xxxxxxxxxxxx> wrote: > On 08/05/18 01:34 PM, Alex Williamson wrote: > > They are not so unrelated, see the ACS Direct Translated P2P > > capability, which in fact must be implemented by switch downstream > > ports implementing ACS and works specifically with ATS. This appears to > > be the way the PCI SIG would intend for P2P to occur within an IOMMU > > managed topology, routing pre-translated DMA directly between peer > > devices while requiring non-translated requests to bounce through the > > IOMMU. Really, what's the value of having an I/O virtual address space > > provided by an IOMMU if we're going to allow physical DMA between > > downstream devices, couldn't we just turn off the IOMMU altogether? Of > > course ATS is not without holes itself, basically that we trust the > > endpoint's implementation of ATS implicitly. Thanks, > > I agree that this is what the SIG intends, but I don't think hardware > fully supports this methodology yet. The Direct Translated capability > just requires switches to forward packets that have the AT request type > set. It does not require them to do the translation or to support ATS > such that P2P requests can be translated by the IOMMU. I expect this is > so that an downstream device can implement ATS and not get messed up by > an upstream switch that doesn't support it. Well, I'm a bit confused, this patch series is specifically disabling ACS on switches, but per the spec downstream switch ports implementing ACS MUST implement direct translated P2P. So it seems the only potential gap here is the endpoint, which must support ATS or else there's nothing for direct translated P2P to do. The switch port plays no part in the actual translation of the request, ATS on the endpoint has already cached the translation and is now attempting to use it. For the switch port, this only becomes a routing decision, the request is already translated, therefore ACS RR and EC can be ignored to perform "normal" (direct) routing, as if ACS were not present. It would be a shame to go to all the trouble of creating this no-ACS mode to find out the target hardware supports ATS and should have simply used it, or we should have disabled the IOMMU altogether, which leaves ACS disabled. Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
