On 09/28/2022, Serge Semin wrote: > On Mon, Sep 26, 2022 at 11:08:18PM +0000, William McVicker wrote: > > On 09/26/2022, Serge Semin wrote: > > > On Mon, Sep 26, 2022 at 04:31:28PM +0200, Christoph Hellwig wrote: > > > > On Mon, Sep 26, 2022 at 03:49:24PM +0300, Serge Semin wrote: > > > > > @Christoph, @Marek, @Bjorn, @Rob could you please join to the > > > > > DMA-mask related discussion. @Lorenzo can't decide which driver should > > > > > initialize the device DMA-mask. > > > > > > > > > > > The driver that does the actual DMA mapping or allocation functions > > > > need to set it. But even with your comments on the questions I'm > > > > still confused what struct device you are even talking about. Can > > > > you explain this a bit better? > > > > > > We are talking about the DW PCIe Root Port controller with DW eDMA engine > > > embedded. It' simplified structure can be represented as follows: > > > > > > +---------------+ +--------+ > > > | System memory | | CPU(s) | > > > +---------------+ +--------+ > > > ^ | | ^ > > > | ... System bus ... | > > > ... | | ... > > > | v v | > > > +------------+------+--------+----------+------+ > > > | DW PCIe RP | AXI-m| | AXI-s/DBI| | > > > | +------+ +----------+ | > > > | ^ ^ | | > > > | +------+----+ | CSRs | > > > | v v v | > > > | +-------+ +---------+ +----------+ | > > > | | eDMA | | in-iATU | | out-iATU | | > > > | +-------+ +---------+ +----------+ | > > > | ^ ^ ^ | > > > | +--------+--+---+-------+ | > > > +------------------| PIPE |--------------------+ > > > +------+ > > > | ^ > > > v | > > > PCIe bus > > > > > > The DW PCIe controller device is instantiated as a platform device > > > defined in the system DT source file. The device is probed by the > > > DW PCIe low-level driver, which after the platform-specific setups > > > initiates the generic DW PCIe host-controller registration. On the way > > > of that procedure the DW PCIe core tries to auto-detect the DW eDMA > > > engine availability. If the engine is found, the DW eDMA probe method > > > is called in order to register the DMA-engine device. After that the > > > PCIe host bridge is registered. Both the PCIe host-bridge and > > > DMA-engine devices will have the DW PCIe platform device as parent. > > > > > > Getting back to the sketch above. Here is a short description of the > > > content: > > > 1. DW eDMA is capable of performing the data transfers from/to System > > > memory to/from PCIe bus memory. > > > 2. in-iATU is the Inbound Address Translation Unit, which is > > > responsible for the PCIe bus peripheral devices to access the system > > > memory. The "dma-ranges" DT-property is used to initialize the > > > PCIe<->Sys memory mapping. (@William note the In-iATU setup doesn't > > > affect the eDMA transfers.) > > > 3. out-iATU is responsible for the CPU(s) to access the PCIe bus > > > peripheral devices memory/cfg-space. > > > > > > So eDMA and in-iATU are using the same AXI-master interface to access > > > the system memory. Thus the DMAable memory capability is the same for > > > both of them (Though in-iATU may have some specific mapping based on > > > the "dma-ranges" DT-property setup). Neither DW eDMA nor DW PCIe Root > > > Port CSRs region have any register to auto-detect the AXI-m interface > > > address bus width. It's selected during the IP-core synthesize and is > > > platform-specific. The question is: "What driver/code is supposed to > > > set the DMA-mask of the DW PCIe platform device?" Seeing the parental > > > platform device is used to perform the memory-mapping for both DW eDMA > > > clients and PCIe-bus peripheral device drivers, and seeing the AXI-m > > > interface parameters aren't auto-detectable and are platform-specific, > > > the only place it should be done in is the DW PCIe low-level device > > > driver. I don't really see any alternative... What is your opinion? > > > > > > -Sergey > > > > > I believe this eDMA implementation is new for an upstream DW PCIe device > > driver, right? If so, this will require some refactoring of the DMA mask code, > > but you need to also make sure you don't break the MSI target address use case > > that prompted this 32-bit DMA mask change -- [1]. > > As far as I can see the commit > https://lore.kernel.org/all/20201117165312.25847-1-vidyas@xxxxxxxxxx/ > isn't marked as fixes or whatever. If so it gets to be pointless due to this > https://elixir.bootlin.com/linux/latest/source/drivers/of/platform.c#L183 > and this > https://elixir.bootlin.com/linux/latest/source/drivers/base/platform.c#L529 > and seeing none of the DW PCIe RP/EP platform drivers change the > device DMA-mask of the being probed platform device. So the mask must > have been of 32-bits anyway even without that commit. > > Moreover as Rob already told you here > https://lore.kernel.org/all/CAL_JsqJh=d-B51b6yPBRq0tOwbChN=AFPr-a19U1QdQZAE7c1A@xxxxxxxxxxxxxx/ > and I mentioned in my response here > https://lore.kernel.org/linux-pci/20220912000211.ct6asuhhmnatje5e@mobilestation/ > DW PCie MSI TLPs never reach the system memory. The TLP PCIe-bus target > address is checked in the host bridge. If it matches to the one > initialized in the iMSI-RX engine CSRs the MSI IRQ will be raised. > None system memory IO will be actually performed. Thus changing the > device DMA-capability due to something which actually doesn't cause > any DMA at the very least inappropriate. Thanks for pointing out the DMA mask references during platform device allocation. I wasn't aware of that. However, I still have issues with using ZONE_DMA32. See comments on how we can address this here: https://lore.kernel.org/linux-pci/YzSJ2ioEeRhHC6zn@xxxxxxxxxx/ > > The last but not least changing the DMA-mask in the common code which > isn't aware of the device/platform capability is also at the very least > inappropriate. > > > My changes were directly > > related to allowing the DW PCIe device driver to fallback to a 64-bit DMA mask > > for the MSI target address if there are no 32-bit allocations available. For > > that use-case, using a 32-bit mask doesn't have any perf impact here since > > there is no actual DMAs happening. > > Regarding your changes. I'll give you my comments in that thread, but > here is a short summary. One more time. There is no actually DMA > performed on MSI due to the way the iMSI-RX works. So setting the > device DMA-mask based on that is inappropriate. Secondly the coherent > memory might be very expensive on some platforms > (see Documentation/core-api/dma-api.rst). And it's on MIPS32 for > instance. Thus using dma_alloc_coherent() > for something other than for real DMA is also inappropriate. What > should have been done instead: > 1. Drop any dma_set_mask*() invocations. I'm fine with this, but others will need to approve of that. > 1. Preserve the alloc_page() method usage. > 2. Pass GFP_DMA32 to the alloc_page() function only if > PCI_MSI_FLAGS_64BIT is set. > > The suggestion above is the best choice seeing we can't reserve some > part of the PCI-bus memory without allocating the real system memory > behind as @Robin noted here in the last paragraph: > https://lore.kernel.org/linux-pci/1e63a581-14ae-b4b5-a5bf-ca8f09c33af6@xxxxxxx/ I'm not okay with this as it re-introduces the dependency on ZONE_DMA32. I responded with more details here with regards to why and how we can work around the ARCH issues with dma_alloc_coherent(): https://lore.kernel.org/linux-pci/YzSJ2ioEeRhHC6zn@xxxxxxxxxx/ Thanks, Will > > -Sergey > > > > > Would it be possible for the DW PCIe device driver to set a capabilities flag > > that the PCIe host controller can read and set the mask accordingly. This way > > you don't need to go fix up any drivers that require a 32-bit DMA'able address > > for the MSI target address. For example, I see several of the PCI capability > > features have 64-bit flags, e.g. PCI_MSI_FLAGS_64BIT and PCI_X_STATUS_64BIT. If > > not, then you're going to have to re-work the host controller driver and DW > > PCIe device drivers that require a 32-bit MSI target address. > > > > [1] https://lore.kernel.org/all/20201117165312.25847-1-vidyas@xxxxxxxxxx/ > > > > Thanks, > > Will > >
