On Wed, Jun 26, 2024 at 01:26:27AM -0700, Shyam Saini wrote: > Hi Serge, > > Apologies for delayed response. > > > > Hi Manivannan, > > > > > > >> Before this change, the dwc PCIe driver configures only 1 ATU region, > > > >> which is sufficient for the devices with PCIe memory <= 4GB. However, > > > >> the driver probe fails when device uses more than 4GB of pcie memory. > > > >> > > > > Something is not clear... This commit message implies that the driver used to > > > > work on your hardware (you haven't mentioned which one it is) and broken by the > > > > commit from Sergey. > > > > > > sorry for any confusion, the driver use to work in v5.10 kernel, with v6.0 kernel it > > > fails to probe with following messages: > > > layerscape-pcie xx00000.pcie: Failed to set MEM range ... > > > layerscape-pcie: probe of xx00000.pcie failed with error -22 > > > > > > By tracing code, I found that the probe was failing on [1] this check, > > > which was added in [2] upstream commit from Serge and finally, the ATU upper bound limit was > > > set to 4G in [3] commit > > > > > > pci driver probe succeeds either when > > > 1) I remove [1] check > > > 2) or by setting the ATU limit to the size of Mem64 range (my original patch [4]) > > > > > > > As per Sergey's commit, we auto detect the dw_pcie::region_limit. If the IP is < > > > > 4.60, then the limit is 4G, otherwise depends on CX_ATU_MAX_REGION_SIZE set in > > > > hw. > > > > > > > > I couldn't find any reference of CX_ATU_MAX_REGION_SIZE in my PCIe TRM, perhaps because it > > > is older than v4.60 > > > > Please find the line containing "iATU: unroll " in the boot log and > > layerscape-pcie xx00000.pcie: iATU: unroll F, 256 ob, 24 ib, align 4K, limit 4G Omg, 256 outbound iATU regions. It's at most _1TB_ PCIe memory mapping. The hardware engineers much have thoroughly considered all the possible controller use-cases.) > > > copy it here. Also please provide a content of the dw_pcie::version > > field _after_ the dw_pcie_version_detect() method is executed. > version is set as 0 > > i see that my driver always has [1] max = 0 > While I am awaiting official info about CX_ATU_MAX_REGION_SIZE, but I think CX_ATU_MAX_REGION_SIZE may not exist It's likely so. The parameter was introduced in IP-core v4.60a. Based on that and not having the PCIE_VERSION_NUMBER_OFF register your IP-core must be older than 4.60a. > > > > > > > > So if your IP is < 4.60, you cannot map > 4GB of outbound memory anyway. But if > > > > it is > 4.60, you shouldn't see the failure that you reported for > 4G space > > > > (well you can see the failure if the limit is less than the region size). In the > > > > previous thread you mentioned that dw_pcie::region_limit is set to 4G. So how > > > > come your driver was working previously? > > > > > > > > The PCIe IP is from Synopsys and is older than 4.60, > > > > If you know what the actual IP-core version is and it's older than > > 4.70a, then it's better to pre-define the version in the > > drivers/pci/controller/dwc/pci-layerscape.c probe procedure (like it's > > done in drivers/pci/controller/dwc/pci-keystone.c, > > drivers/pci/controller/dwc/pcie-bt1.c). > > sure, I will look into that Great! Thanks. > > > > the board is from Freescale LX2 > > > family. > > > > I failed to find the LX2 PCIe controller support in the > > drivers/pci/controller/dwc/pci-layerscape.c driver. AFAICS the > > drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c driver is responsible > > for working with the LX2 PCIe host. Am I missing something? > > driver is drivers/pci/controller/dwc/pci-layerscape.c > device-tree compatible string is "fsl,ls2088a-pcie" Got it. > > > Anyway I've checked all the DT-nodes defined for the Freescale Layerscape > > PCIe host controllers. None of them have the PCIe ranges defined with > > the window size greater than 4G (actually all of them of the 1G size). > > So the next question is: what DT source have you been using on your > > platform and what is the DT-node defined for the PCIe host controller > > in there? > > It is internal and custom pcie range. It is not present in any upstream boards dts files. > > > > The board uses drivers/pci/controller/dwc/pci-layerscape.c driver > > > Given PCIe IP is older than 4.60, I am not sure why it was working earlier for a memory range larger than 4G > > > Highly appreciate your guidance and help on this. > > > > It has been working earlier because the kernel 5.10 didn't have > > PCI ranges check for not exceeding the iATU regions maximum limit. > > true > > > But once again, the DW PCIe Root-port driver has been designed to > > allocate a _single_ iATU region for each PCIe memory ranges. Thus if the > > ranges exceeds the maximum limit, the mapping won't work for the > > addresses greater than the maximum limit. I've already explained it > > here: > > https://lore.kernel.org/linux-pci/hxluc6qth4temdyxloekbhoy4iielyvxmmhp3u47qwtcxb5t5v@v5hdzvqmrsyv > > > > Moreover the IP-core databook not only explicitly prohibits to define the iATU > > ranges greater than 4GB on the DW PCIe device older than 4.60a (text > > from v4.21a databook), but has more strict constraint of the regions > > not crossing the 4GB boundary: > > "The upper 32 bits of the Target Address register always forms the > > upper 32 bits of the translated address because: > > ■ The maximum region size is 4 GB. > > ■ A region must not cross a 4 GB boundary." > > > > (It's obvious though since the iATU Limit Address register is of the > > 32-bit size on the old controllers.) > > > > So you either need to allocate several iATU regions to cover the > > requested PCIe ranges, or fix the PCIe controller DT-node to having the > > ranges not exceeding the maximum limit. > > we have requirement for memory ranmge greater than 4G, so it seems range can't be changed. > > I am looking into your other suggestions, highly appreciate your feedback Ok. Thanks. -Serge(y) > > Thanks, > Shyam > > [1] https://elixir.bootlin.com/linux/v6.10-rc5/source/drivers/pci/controller/dwc/pcie-designware.c#L832 >
