Hi Shyam On Thu, Jun 13, 2024 at 03:47:47PM -0700, Shyam Saini wrote: > Hi Manivannan, > > >> Before this change, the dwc PCIe driver configures only 1 ATU region, > >> which is sufficient for the devices with PCIe memory <= 4GB. However, > >> the driver probe fails when device uses more than 4GB of pcie memory. > >> > > Something is not clear... This commit message implies that the driver used to > > work on your hardware (you haven't mentioned which one it is) and broken by the > > commit from Sergey. > > sorry for any confusion, the driver use to work in v5.10 kernel, with v6.0 kernel it > fails to probe with following messages: > layerscape-pcie xx00000.pcie: Failed to set MEM range ... > layerscape-pcie: probe of xx00000.pcie failed with error -22 > > By tracing code, I found that the probe was failing on [1] this check, > which was added in [2] upstream commit from Serge and finally, the ATU upper bound limit was > set to 4G in [3] commit > > pci driver probe succeeds either when > 1) I remove [1] check > 2) or by setting the ATU limit to the size of Mem64 range (my original patch [4]) > > > As per Sergey's commit, we auto detect the dw_pcie::region_limit. If the IP is < > > 4.60, then the limit is 4G, otherwise depends on CX_ATU_MAX_REGION_SIZE set in > > hw. > > I couldn't find any reference of CX_ATU_MAX_REGION_SIZE in my PCIe TRM, perhaps because it > is older than v4.60 Please find the line containing "iATU: unroll " in the boot log and copy it here. Also please provide a content of the dw_pcie::version field _after_ the dw_pcie_version_detect() method is executed. > > > So if your IP is < 4.60, you cannot map > 4GB of outbound memory anyway. But if > > it is > 4.60, you shouldn't see the failure that you reported for > 4G space > > (well you can see the failure if the limit is less than the region size). In the > > previous thread you mentioned that dw_pcie::region_limit is set to 4G. So how > > come your driver was working previously? > > The PCIe IP is from Synopsys and is older than 4.60, If you know what the actual IP-core version is and it's older than 4.70a, then it's better to pre-define the version in the drivers/pci/controller/dwc/pci-layerscape.c probe procedure (like it's done in drivers/pci/controller/dwc/pci-keystone.c, drivers/pci/controller/dwc/pcie-bt1.c). > the board is from Freescale LX2 > family. I failed to find the LX2 PCIe controller support in the drivers/pci/controller/dwc/pci-layerscape.c driver. AFAICS the drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c driver is responsible for working with the LX2 PCIe host. Am I missing something? Anyway I've checked all the DT-nodes defined for the Freescale Layerscape PCIe host controllers. None of them have the PCIe ranges defined with the window size greater than 4G (actually all of them of the 1G size). So the next question is: what DT source have you been using on your platform and what is the DT-node defined for the PCIe host controller in there? > The board uses drivers/pci/controller/dwc/pci-layerscape.c driver > Given PCIe IP is older than 4.60, I am not sure why it was working earlier for a memory range larger than 4G > Highly appreciate your guidance and help on this. It has been working earlier because the kernel 5.10 didn't have PCI ranges check for not exceeding the iATU regions maximum limit. But once again, the DW PCIe Root-port driver has been designed to allocate a _single_ iATU region for each PCIe memory ranges. Thus if the ranges exceeds the maximum limit, the mapping won't work for the addresses greater than the maximum limit. I've already explained it here: https://lore.kernel.org/linux-pci/hxluc6qth4temdyxloekbhoy4iielyvxmmhp3u47qwtcxb5t5v@v5hdzvqmrsyv Moreover the IP-core databook not only explicitly prohibits to define the iATU ranges greater than 4GB on the DW PCIe device older than 4.60a (text from v4.21a databook), but has more strict constraint of the regions not crossing the 4GB boundary: "The upper 32 bits of the Target Address register always forms the upper 32 bits of the translated address because: ■ The maximum region size is 4 GB. ■ A region must not cross a 4 GB boundary." (It's obvious though since the iATU Limit Address register is of the 32-bit size on the old controllers.) So you either need to allocate several iATU regions to cover the requested PCIe ranges, or fix the PCIe controller DT-node to having the ranges not exceeding the maximum limit. -Serge(y) > > Best regards, > Shyam > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware.c?h=v6.10-rc3#n480 > [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/pci/controller/dwc?h=v6.10-rc3&id=edf408b946d37cc70fbf0db6ab85bbf67dae1894 > [3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/pci/controller/dwc?h=v6.10-rc3&id=89473aa9ab261948ed13b16effe841a675efed77 > [4] https://lore.kernel.org/linux-pci/20240523001046.1413579-1-shyamsaini@xxxxxxxxxxxxxxxxxxx/ > > > >
