On Mon, Sep 26, 2022 at 11:08:18PM +0000, William McVicker wrote: > On 09/26/2022, Serge Semin wrote: > > On Mon, Sep 26, 2022 at 04:31:28PM +0200, Christoph Hellwig wrote: > > > On Mon, Sep 26, 2022 at 03:49:24PM +0300, Serge Semin wrote: > > > > @Christoph, @Marek, @Bjorn, @Rob could you please join to the > > > > DMA-mask related discussion. @Lorenzo can't decide which driver should > > > > initialize the device DMA-mask. > > > > > > > > The driver that does the actual DMA mapping or allocation functions > > > need to set it. But even with your comments on the questions I'm > > > still confused what struct device you are even talking about. Can > > > you explain this a bit better? > > > > We are talking about the DW PCIe Root Port controller with DW eDMA engine > > embedded. It' simplified structure can be represented as follows: > > > > +---------------+ +--------+ > > | System memory | | CPU(s) | > > +---------------+ +--------+ > > ^ | | ^ > > | ... System bus ... | > > ... | | ... > > | v v | > > +------------+------+--------+----------+------+ > > | DW PCIe RP | AXI-m| | AXI-s/DBI| | > > | +------+ +----------+ | > > | ^ ^ | | > > | +------+----+ | CSRs | > > | v v v | > > | +-------+ +---------+ +----------+ | > > | | eDMA | | in-iATU | | out-iATU | | > > | +-------+ +---------+ +----------+ | > > | ^ ^ ^ | > > | +--------+--+---+-------+ | > > +------------------| PIPE |--------------------+ > > +------+ > > | ^ > > v | > > PCIe bus > > > > The DW PCIe controller device is instantiated as a platform device > > defined in the system DT source file. The device is probed by the > > DW PCIe low-level driver, which after the platform-specific setups > > initiates the generic DW PCIe host-controller registration. On the way > > of that procedure the DW PCIe core tries to auto-detect the DW eDMA > > engine availability. If the engine is found, the DW eDMA probe method > > is called in order to register the DMA-engine device. After that the > > PCIe host bridge is registered. Both the PCIe host-bridge and > > DMA-engine devices will have the DW PCIe platform device as parent. > > > > Getting back to the sketch above. Here is a short description of the > > content: > > 1. DW eDMA is capable of performing the data transfers from/to System > > memory to/from PCIe bus memory. > > 2. in-iATU is the Inbound Address Translation Unit, which is > > responsible for the PCIe bus peripheral devices to access the system > > memory. The "dma-ranges" DT-property is used to initialize the > > PCIe<->Sys memory mapping. (@William note the In-iATU setup doesn't > > affect the eDMA transfers.) > > 3. out-iATU is responsible for the CPU(s) to access the PCIe bus > > peripheral devices memory/cfg-space. > > > > So eDMA and in-iATU are using the same AXI-master interface to access > > the system memory. Thus the DMAable memory capability is the same for > > both of them (Though in-iATU may have some specific mapping based on > > the "dma-ranges" DT-property setup). Neither DW eDMA nor DW PCIe Root > > Port CSRs region have any register to auto-detect the AXI-m interface > > address bus width. It's selected during the IP-core synthesize and is > > platform-specific. The question is: "What driver/code is supposed to > > set the DMA-mask of the DW PCIe platform device?" Seeing the parental > > platform device is used to perform the memory-mapping for both DW eDMA > > clients and PCIe-bus peripheral device drivers, and seeing the AXI-m > > interface parameters aren't auto-detectable and are platform-specific, > > the only place it should be done in is the DW PCIe low-level device > > driver. I don't really see any alternative... What is your opinion? > > > > -Sergey > > I believe this eDMA implementation is new for an upstream DW PCIe device > driver, right? If so, this will require some refactoring of the DMA mask code, > but you need to also make sure you don't break the MSI target address use case > that prompted this 32-bit DMA mask change -- [1]. As far as I can see the commit https://lore.kernel.org/all/20201117165312.25847-1-vidyas@xxxxxxxxxx/ isn't marked as fixes or whatever. If so it gets to be pointless due to this https://elixir.bootlin.com/linux/latest/source/drivers/of/platform.c#L183 and this https://elixir.bootlin.com/linux/latest/source/drivers/base/platform.c#L529 and seeing none of the DW PCIe RP/EP platform drivers change the device DMA-mask of the being probed platform device. So the mask must have been of 32-bits anyway even without that commit. Moreover as Rob already told you here https://lore.kernel.org/all/CAL_JsqJh=d-B51b6yPBRq0tOwbChN=AFPr-a19U1QdQZAE7c1A@xxxxxxxxxxxxxx/ and I mentioned in my response here https://lore.kernel.org/linux-pci/20220912000211.ct6asuhhmnatje5e@mobilestation/ DW PCie MSI TLPs never reach the system memory. The TLP PCIe-bus target address is checked in the host bridge. If it matches to the one initialized in the iMSI-RX engine CSRs the MSI IRQ will be raised. None system memory IO will be actually performed. Thus changing the device DMA-capability due to something which actually doesn't cause any DMA at the very least inappropriate. The last but not least changing the DMA-mask in the common code which isn't aware of the device/platform capability is also at the very least inappropriate. > My changes were directly > related to allowing the DW PCIe device driver to fallback to a 64-bit DMA mask > for the MSI target address if there are no 32-bit allocations available. For > that use-case, using a 32-bit mask doesn't have any perf impact here since > there is no actual DMAs happening. Regarding your changes. I'll give you my comments in that thread, but here is a short summary. One more time. There is no actually DMA performed on MSI due to the way the iMSI-RX works. So setting the device DMA-mask based on that is inappropriate. Secondly the coherent memory might be very expensive on some platforms (see Documentation/core-api/dma-api.rst). And it's on MIPS32 for instance. Thus using dma_alloc_coherent() for something other than for real DMA is also inappropriate. What should have been done instead: 1. Drop any dma_set_mask*() invocations. 1. Preserve the alloc_page() method usage. 2. Pass GFP_DMA32 to the alloc_page() function only if PCI_MSI_FLAGS_64BIT is set. The suggestion above is the best choice seeing we can't reserve some part of the PCI-bus memory without allocating the real system memory behind as @Robin noted here in the last paragraph: https://lore.kernel.org/linux-pci/1e63a581-14ae-b4b5-a5bf-ca8f09c33af6@xxxxxxx/ -Sergey > > Would it be possible for the DW PCIe device driver to set a capabilities flag > that the PCIe host controller can read and set the mask accordingly. This way > you don't need to go fix up any drivers that require a 32-bit DMA'able address > for the MSI target address. For example, I see several of the PCI capability > features have 64-bit flags, e.g. PCI_MSI_FLAGS_64BIT and PCI_X_STATUS_64BIT. If > not, then you're going to have to re-work the host controller driver and DW > PCIe device drivers that require a 32-bit MSI target address. > > [1] https://lore.kernel.org/all/20201117165312.25847-1-vidyas@xxxxxxxxxx/ > > Thanks, > Will >
