On 08/09/2018 03:40 PM, Jiang, Dave wrote:
-----Original Message-----
From: linux-pci-owner@xxxxxxxxxxxxxxx [mailto:linux-pci-owner@xxxxxxxxxxxxxxx] On Behalf Of Kit Chow
Sent: Thursday, August 9, 2018 2:48 PM
To: Logan Gunthorpe <logang@xxxxxxxxxxxx>; Eric Pilmore <epilmore@xxxxxxxxxx>; Bjorn Helgaas <helgaas@xxxxxxxxxx>
Cc: linux-pci@xxxxxxxxxxxxxxx; David Woodhouse <dwmw2@xxxxxxxxxxxxx>; Alex Williamson <alex.williamson@xxxxxxxxxx>;
iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx
Subject: Re: IOAT DMA w/IOMMU
On 08/09/2018 02:11 PM, Logan Gunthorpe wrote:
On 09/08/18 02:57 PM, Kit Chow wrote:
On 08/09/2018 01:11 PM, Logan Gunthorpe wrote:
On 09/08/18 01:47 PM, Kit Chow wrote:
I haven't tested this scenario but my guess would be that IOAT would
indeed go through the IOMMU and the PCI BAR address would need to be
properly mapped into the IOAT's IOVA. The fact that you see DMAR errors
is probably a good indication that this is the case. I really don't know
why you'd want to DMA something without mapping it.
I have experimented with changing ntb_async_tx_submit to dma_map the PCI
BAR
address. With this, I get a different DMAR error:
What code did you use to do this?
If you mean version of linux, it is 4.15.7. Or specific dma_map call, I
believe it was dma_map_single.
I mean the complete code you use to do the mapping so we can see if it's
correct. dma_map_single() seems like an odd choice, I expected to see
dma_map_resource().
Thanks for the suggestion! Will try out dma_map_resource and report back.
Kit
Kit, I was able to try this on the Skylake Xeon platform with Intel NTB and ioatdma and I did not encounter any issues. Unfortunately I do not have my pre-Skylake platform to test anymore to verify.
We are expecting to get skylakes within a couple of weeks and I'll have
a chance to give it a go on there.
Based on Logan's comments, I am very hopeful that the dma_map_resource
will make things work on the older platforms...
Thanks
Kit
DMAR: [DMA Write] Request device [07:00.4] fault addr ffd00000
[fault reason 12] non-zero reserved fields in PTE
Also, what device corresponds to 07:00.4 on your system?
I believe 07.00.4 was the PLX dma device. I get the same error with ioat.
Using the mapping with the PLX dma device likely converts it from a pure
P2P request to one where the TLPs pass through the IOMMU. So the fact
that you get the same error with both means IOAT almost certainly goes
through the IOMMU and there's something wrong with the mapping setup.
Logan
