On Mon, May 13, 2024 at 04:53:50PM -0500, Bjorn Helgaas wrote:
> On Thu, Mar 28, 2024 at 02:20:41PM +0530, Siddharth Vadapalli wrote:
> > In the process of converting .scan_bus() callbacks to .add_bus(), the
> > ks_pcie_v3_65_scan_bus() function was changed to ks_pcie_v3_65_add_bus().
> > The .scan_bus() method belonged to ks_pcie_host_ops which was specific
> > to controller version 3.65a, while the .add_bus() method had been added
> > to ks_pcie_ops which is shared between the controller versions 3.65a and
> > 4.90a. Neither the older ks_pcie_v3_65_scan_bus() method, nor the newer
> > ks_pcie_v3_65_add_bus() method is applicable to the controller version
> > 4.90a which is present in AM654x SoCs.
> >
> > Thus, as a fix, remove "ks_pcie_v3_65_add_bus()" and move its contents
> > to the .msi_init callback "ks_pcie_msi_host_init()" which is specific to
> > the 3.65a controller.
> >
> > Fixes: 6ab15b5e7057 ("PCI: dwc: keystone: Convert .scan_bus() callback to use add_bus")
> > Suggested-by: Serge Semin <fancer.lancer@xxxxxxxxx>
> > Suggested-by: Bjorn Helgaas <helgaas@xxxxxxxxxx>
> > Suggested-by: Niklas Cassel <cassel@xxxxxxxxxx>
> > Reviewed-by: Niklas Cassel <cassel@xxxxxxxxxx>
> > Signed-off-by: Siddharth Vadapalli <s-vadapalli@xxxxxx>
>
> Thanks for splitting this into two patches. Krzysztof has applied
> both to pci/controller/keystone and we hope to merge them for v6.10.
>
> I *would* like the commit log to be at a little higher level if
> possible. Right now it's a detailed description at the level of the
> code edits, but it doesn't say *why* we want this change.
>
> I think the first cut at this was
> https://lore.kernel.org/linux-pci/20231011123451.34827-1-s-vadapalli@xxxxxx/t/#u,
> which mentioned Completion Timeouts during MSI-X configuration and 45
> second delays during boot.
>
> IIUC, prior to 6ab15b5e7057, ks_pcie_v3_65_scan_bus() initialized BAR
> 0 and was only used for v3.65a devices. 6ab15b5e7057 renamed it to
> ks_pcie_v3_65_add_bus() and called it for both v3.65a and v4.90a.
>
> I think the problem is that in the current code, the
> ks_pcie_ops.add_bus() method (ks_pcie_v3_65_add_bus()) is used for all
> devices (both v3.65a and v4.90a). So I guess doing the BAR 0 setup on
> v4.90a broke something there?
BAR0 was set to a different value on AM654x SoC which has the v4.90a
controller, which is identical to what is set even for the v3.65a
controller. The difference is that BAR0 is programmed to a different
value for enabling inbound MSI writes on top of the common configuration
performed for BAR0.
Common configuration for BAR0:
ks_pcie_probe
dw_pcie_host_init
dw_pcie_setup_rc
...
/* Setup RC BARs */
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
...
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0);
...
MSI specific configuration of BAR0 performed after the common
configuration via the ks_pcie_v3_65_scan_bus() callback:
/* Configure and set up BAR0 */
ks_pcie_set_dbi_mode(ks_pcie);
/* Enable BAR0 */
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 1);
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, SZ_4K - 1);
ks_pcie_clear_dbi_mode(ks_pcie);
/*
* For BAR0, just setting bus address for inbound writes (MSI) should
* be sufficient. Use physical address to avoid any conflicts.
*/
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, ks_pcie->app.start);
The above configuration of BAR0 shouldn't be performed for AM654x SoC.
While I am not certain, the timeouts are probably a result of the BAR
being programmed to a wrong value which results in a "no match" outcome.
>
> I'm not quite clear on the mechanism, but it would be helpful to at
> least know what's wrong and on what platform. E.g., currently v4.90
> suffers Completion Timeouts and 45 second boot delays? And this patch
> fixes that?
Yes, the Completion Timeouts cause the 45 second boot delays and this
patch fixes that.
Regards,
Siddharth.
