On 2022-12-15 09:27, Serge Semin wrote:
Hi Christoph
On Wed, Dec 14, 2022 at 11:13:41PM -0800, Christoph Hellwig wrote:
On Thu, Dec 15, 2022 at 02:53:03AM +0300, Serge Semin wrote:
DW PCIe Root Ports and End-points can be equipped with the DW eDMA engine.
In that case it is critical to have the platform device pre-initialized
with a valid DMA-mask so the drivers using the eDMA-engine would be able
to allocate the DMA-able buffers. The MSI-capable data requires to be
allocated from the lowest 4GB region. Since that procedure implies the
DMA-mask change we need to restore the mask set by the low-level drivers
after the MSI-data allocation is done.
You can't change the DMA mask when there are existing allocations.
Em, what do you guys suggest for the DW PCIe devices with the embedded
DMA-engine then? To live forever with the SWIOTLBs? I can't drop the
DMA-mask update due to this commit 423511ec23e2 ("PCI: dwc: Drop
dependency on ZONE_DMA32") and I can't change the mask after it's
updated. Note it's updated for the memory allocation to which actually
no DMA will be performed, see
https://lore.kernel.org/linux-pci/20220825185026.3816331-2-willmcvicker@xxxxxxxxxx/.
My patches imply adding the real DMA operations support.
We've discussed this a lot with Robin in various threads and I thought
a workable solution was found. I was going to update the mask in
another place, but basically it would still mean to have first setting
the 32-bit mask here, and then change it to 64-bit one in the
framework of the DW eDMA driver.
So to speak I don't see a proper way out from the situation. Nothing I
suggested was accepted and now we'll have to live with the SWIOTLBs
used for the memory above 4GB. So please suggest a workable solution
then. We need the next things:
1. Somehow preserve a single DWORD of the PCIe bus memory for the
iMSI-RX engine. (That's what is currently done the
dw_pcie_msi_host_init() method by allocating the coherent memory.)
2. Set the actual DMA-mask to the DW PCIe platform device so the
DMA-engine clients would be able to allocate actually DMA-able memory.
@Robin, please join the discussion.
Basically just don't touch the coherent mask. The eDMA drivers can still
set the streaming mask to something larger, and that's the one that's
going to matter for most dmaengine clients anyway. Even if someone does
call dma_alloc_coherent() for their eDMA channel, it's not going to make
much practical difference if that has to come from a DMA zone, unless
the system is under severe memory pressure anyway.
Thanks,
Robin.
