Hi Serge, Apologies for delayed response. > > Hi Manivannan, > > > > >> Before this change, the dwc PCIe driver configures only 1 ATU region, > > >> which is sufficient for the devices with PCIe memory <= 4GB. However, > > >> the driver probe fails when device uses more than 4GB of pcie memory. > > >> > > > Something is not clear... This commit message implies that the driver used to > > > work on your hardware (you haven't mentioned which one it is) and broken by the > > > commit from Sergey. > > > > sorry for any confusion, the driver use to work in v5.10 kernel, with v6.0 kernel it > > fails to probe with following messages: > > layerscape-pcie xx00000.pcie: Failed to set MEM range ... > > layerscape-pcie: probe of xx00000.pcie failed with error -22 > > > > By tracing code, I found that the probe was failing on [1] this check, > > which was added in [2] upstream commit from Serge and finally, the ATU upper bound limit was > > set to 4G in [3] commit > > > > pci driver probe succeeds either when > > 1) I remove [1] check > > 2) or by setting the ATU limit to the size of Mem64 range (my original patch [4]) > > > > > As per Sergey's commit, we auto detect the dw_pcie::region_limit. If the IP is < > > > 4.60, then the limit is 4G, otherwise depends on CX_ATU_MAX_REGION_SIZE set in > > > hw. > > > > > I couldn't find any reference of CX_ATU_MAX_REGION_SIZE in my PCIe TRM, perhaps because it > > is older than v4.60 > > Please find the line containing "iATU: unroll " in the boot log and layerscape-pcie xx00000.pcie: iATU: unroll F, 256 ob, 24 ib, align 4K, limit 4G > copy it here. Also please provide a content of the dw_pcie::version > field _after_ the dw_pcie_version_detect() method is executed. version is set as 0 i see that my driver always has [1] max = 0 While I am awaiting official info about CX_ATU_MAX_REGION_SIZE, but I think CX_ATU_MAX_REGION_SIZE may not exist > > > > > So if your IP is < 4.60, you cannot map > 4GB of outbound memory anyway. But if > > > it is > 4.60, you shouldn't see the failure that you reported for > 4G space > > > (well you can see the failure if the limit is less than the region size). In the > > > previous thread you mentioned that dw_pcie::region_limit is set to 4G. So how > > > come your driver was working previously? > > > > > The PCIe IP is from Synopsys and is older than 4.60, > > If you know what the actual IP-core version is and it's older than > 4.70a, then it's better to pre-define the version in the > drivers/pci/controller/dwc/pci-layerscape.c probe procedure (like it's > done in drivers/pci/controller/dwc/pci-keystone.c, > drivers/pci/controller/dwc/pcie-bt1.c). sure, I will look into that > > the board is from Freescale LX2 > > family. > > I failed to find the LX2 PCIe controller support in the > drivers/pci/controller/dwc/pci-layerscape.c driver. AFAICS the > drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c driver is responsible > for working with the LX2 PCIe host. Am I missing something? driver is drivers/pci/controller/dwc/pci-layerscape.c device-tree compatible string is "fsl,ls2088a-pcie" > Anyway I've checked all the DT-nodes defined for the Freescale Layerscape > PCIe host controllers. None of them have the PCIe ranges defined with > the window size greater than 4G (actually all of them of the 1G size). > So the next question is: what DT source have you been using on your > platform and what is the DT-node defined for the PCIe host controller > in there? It is internal and custom pcie range. It is not present in any upstream boards dts files. > > The board uses drivers/pci/controller/dwc/pci-layerscape.c driver > > Given PCIe IP is older than 4.60, I am not sure why it was working earlier for a memory range larger than 4G > > Highly appreciate your guidance and help on this. > > It has been working earlier because the kernel 5.10 didn't have > PCI ranges check for not exceeding the iATU regions maximum limit. true > But once again, the DW PCIe Root-port driver has been designed to > allocate a _single_ iATU region for each PCIe memory ranges. Thus if the > ranges exceeds the maximum limit, the mapping won't work for the > addresses greater than the maximum limit. I've already explained it > here: > https://lore.kernel.org/linux-pci/hxluc6qth4temdyxloekbhoy4iielyvxmmhp3u47qwtcxb5t5v@v5hdzvqmrsyv > > Moreover the IP-core databook not only explicitly prohibits to define the iATU > ranges greater than 4GB on the DW PCIe device older than 4.60a (text > from v4.21a databook), but has more strict constraint of the regions > not crossing the 4GB boundary: > "The upper 32 bits of the Target Address register always forms the > upper 32 bits of the translated address because: > ■ The maximum region size is 4 GB. > ■ A region must not cross a 4 GB boundary." > > (It's obvious though since the iATU Limit Address register is of the > 32-bit size on the old controllers.) > > So you either need to allocate several iATU regions to cover the > requested PCIe ranges, or fix the PCIe controller DT-node to having the > ranges not exceeding the maximum limit. we have requirement for memory ranmge greater than 4G, so it seems range can't be changed. I am looking into your other suggestions, highly appreciate your feedback Thanks, Shyam [1] https://elixir.bootlin.com/linux/v6.10-rc5/source/drivers/pci/controller/dwc/pcie-designware.c#L832
