Hi Arnd, On 12 November 2015 09:49, Arnd Bergmann wrote: > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > On 11 November 2015 18:25, LIviu wrote: > > > On Mon, Nov 09, 2015 at 12:32:13PM +0000, Phil Edworthy wrote: > > > > I think you're mixing things a bit or not explaining them very well. Having the > > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > > carry 64-bit addresses. It depends on how they get translated by the host > bridge > > > or its associated ATS block. I can't see why you can't have a setup where > > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > > You just have to be careful on how you setup your mem64 ranges so that > they > > > don't > > > overlap with the 32-bit ranges when translated. > > From a HW point of view I agree that we can setup the PCI host bridge such that > > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't > > this mean that the dma ops used by card drivers has to be provided by our PCI > > host bridge driver so we can apply the translation to those PCI addresses? > > This comes back to my point below about how to do this. Adding a bus notifier > > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > the > > > card is not capable of supporting more than 32-bit addresses. > > If there was infrastructure that checked all parents dma-ranges when the > > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > > the problem. > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > and then calls arch_setup_dma_ops() so the architecture specific code can > enforce the limits in dma_set_mask and pick an appropriate set of dma > operations. The missing part is in the implementation of arch_setup_dma_ops, > which currently happily ignores the base and limit. I don't think it's as simple as that, though I could be wrong! First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. This default is set for the 'platform soc' device. For my own testing I increased this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes boot failure that I haven't looked into. Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before calling of_pci_dma_configure(). I assume it does this on the basis that this is a good default for PCI drivers that don't call dma_set_mask(). So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit this mask. Finally, dma_set_mask_and_coherent() is called from the PCI card driver but it doesn't check the parents dma masks either. Thanks Phil -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
