> -----Original Message----- > From: linux-pci-owner@xxxxxxxxxxxxxxx [mailto:linux-pci-owner@xxxxxxxxxxxxxxx] On Behalf Of Kit Chow > Sent: Thursday, August 9, 2018 2:48 PM > To: Logan Gunthorpe <logang@xxxxxxxxxxxx>; Eric Pilmore <epilmore@xxxxxxxxxx>; Bjorn Helgaas <helgaas@xxxxxxxxxx> > Cc: linux-pci@xxxxxxxxxxxxxxx; David Woodhouse <dwmw2@xxxxxxxxxxxxx>; Alex Williamson <alex.williamson@xxxxxxxxxx>; > iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx > Subject: Re: IOAT DMA w/IOMMU > > > > On 08/09/2018 02:11 PM, Logan Gunthorpe wrote: > > > > On 09/08/18 02:57 PM, Kit Chow wrote: > >> > >> On 08/09/2018 01:11 PM, Logan Gunthorpe wrote: > >>> On 09/08/18 01:47 PM, Kit Chow wrote: > >>>>> I haven't tested this scenario but my guess would be that IOAT would > >>>>> indeed go through the IOMMU and the PCI BAR address would need to be > >>>>> properly mapped into the IOAT's IOVA. The fact that you see DMAR errors > >>>>> is probably a good indication that this is the case. I really don't know > >>>>> why you'd want to DMA something without mapping it. > >>>> I have experimented with changing ntb_async_tx_submit to dma_map the PCI > >>>> BAR > >>>> address. With this, I get a different DMAR error: > >>> What code did you use to do this? > >> If you mean version of linux, it is 4.15.7. Or specific dma_map call, I > >> believe it was dma_map_single. > > I mean the complete code you use to do the mapping so we can see if it's > > correct. dma_map_single() seems like an odd choice, I expected to see > > dma_map_resource(). > Thanks for the suggestion! Will try out dma_map_resource and report back. > > Kit Kit, I was able to try this on the Skylake Xeon platform with Intel NTB and ioatdma and I did not encounter any issues. Unfortunately I do not have my pre-Skylake platform to test anymore to verify. > > >>>> DMAR: [DMA Write] Request device [07:00.4] fault addr ffd00000 > >>>> [fault reason 12] non-zero reserved fields in PTE > >>> Also, what device corresponds to 07:00.4 on your system? > >> I believe 07.00.4 was the PLX dma device. I get the same error with ioat. > > Using the mapping with the PLX dma device likely converts it from a pure > > P2P request to one where the TLPs pass through the IOMMU. So the fact > > that you get the same error with both means IOAT almost certainly goes > > through the IOMMU and there's something wrong with the mapping setup. > > > > Logan
