Hi Manivannan,
> From: Manivannan Sadhasivam, Sent: Saturday, April 22, 2023 11:38 PM
>
> On Tue, Apr 18, 2023 at 09:24:00PM +0900, Yoshihiro Shimoda wrote:
> > Add R-Car Gen4 PCIe Host support. This controller is based on
> > Synopsys DesignWare PCIe.
> >
>
> It is good to add more details about the controller here like retraining etc...
I got it. I'll add such description.
> Also, please justify why you have added some helpers in a separate file. If
> those helpers are going to be used in other drivers now, then it should be
> mentioned here.
>
> NOTE: It may be used in future is not a valid justification.
I'll add such description too.
> > Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@xxxxxxxxxxx>
> > ---
> > drivers/pci/controller/dwc/Kconfig | 9 +
> > drivers/pci/controller/dwc/Makefile | 2 +
> > .../pci/controller/dwc/pcie-rcar-gen4-host.c | 134 +++++++++++++
> > drivers/pci/controller/dwc/pcie-rcar-gen4.c | 187 ++++++++++++++++++
> > drivers/pci/controller/dwc/pcie-rcar-gen4.h | 49 +++++
> > 5 files changed, 381 insertions(+)
> > create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > create mode 100644 drivers/pci/controller/dwc/pcie-rcar-gen4.h
> >
> > diff --git a/drivers/pci/controller/dwc/Kconfig b/drivers/pci/controller/dwc/Kconfig
> > index d29551261e80..eb90e2116e59 100644
> > --- a/drivers/pci/controller/dwc/Kconfig
> > +++ b/drivers/pci/controller/dwc/Kconfig
> > @@ -415,4 +415,13 @@ config PCIE_FU740
> > Say Y here if you want PCIe controller support for the SiFive
> > FU740.
> >
> > +config PCIE_RCAR_GEN4
> > + tristate "Renesas R-Car Gen4 PCIe Host controller"
> > + depends on ARCH_RENESAS || COMPILE_TEST
> > + depends on PCI_MSI
> > + select PCIE_DW_HOST
> > + help
> > + Say Y here if you want PCIe host controller support on R-Car Gen4 SoCs.
> > + This uses the DesignWare core.
> > +
> > endmenu
> > diff --git a/drivers/pci/controller/dwc/Makefile b/drivers/pci/controller/dwc/Makefile
> > index bf5c311875a1..486cf706b53d 100644
> > --- a/drivers/pci/controller/dwc/Makefile
> > +++ b/drivers/pci/controller/dwc/Makefile
> > @@ -26,6 +26,8 @@ obj-$(CONFIG_PCIE_TEGRA194) += pcie-tegra194.o
> > obj-$(CONFIG_PCIE_UNIPHIER) += pcie-uniphier.o
> > obj-$(CONFIG_PCIE_UNIPHIER_EP) += pcie-uniphier-ep.o
> > obj-$(CONFIG_PCIE_VISCONTI_HOST) += pcie-visconti.o
> > +pcie-rcar-gen4-host-drv-objs := pcie-rcar-gen4.o pcie-rcar-gen4-host.o
> > +obj-$(CONFIG_PCIE_RCAR_GEN4) += pcie-rcar-gen4-host-drv.o
> >
> > # The following drivers are for devices that use the generic ACPI
> > # pci_root.c driver but don't support standard ECAM config access.
> > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > new file mode 100644
> > index 000000000000..067fbd2a8d50
> > --- /dev/null
> > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4-host.c
> > @@ -0,0 +1,134 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * PCIe host controller driver for Renesas R-Car Gen4 Series SoCs
> > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > + */
> > +
> > +#include <linux/interrupt.h>
> > +#include <linux/module.h>
> > +#include <linux/of_device.h>
> > +#include <linux/pci.h>
> > +#include <linux/platform_device.h>
> > +
> > +#include "pcie-rcar-gen4.h"
> > +#include "pcie-designware.h"
> > +
> > +static int rcar_gen4_pcie_host_init(struct dw_pcie_rp *pp)
> > +{
> > + struct dw_pcie *dw = to_dw_pcie_from_pp(pp);
> > + struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > + int ret;
> > + u32 val;
> > +
>
> Don't you need to assert perst before starting controller config?
You're correct. Without assert perst, a PCIe device is possible to work,
but this driver should assert it with msleep(100) interval here.
> > + ret = rcar_gen4_pcie_set_device_type(rcar, true, dw->num_lanes);
> > + if (ret < 0)
> > + return ret;
> > +
> > + dw_pcie_dbi_ro_wr_en(dw);
> > +
> > + /*
> > + * According to the databook, we should disable two BARs to avoid
>
> Which databook? Synopsys DWC?
Yes, it's Synopsys DWC. I referred the section 3.5.7.2 "RC Mode" in DWC PCIe Dual Mode Rev.5.20a.
> > + * unnecessary memory assignment during device enumeration.
> > + */
> > + rcar_gen4_pcie_disable_bar(dw, PCI_BASE_ADDRESS_0);
> > + rcar_gen4_pcie_disable_bar(dw, PCI_BASE_ADDRESS_1);
>
> I don't see a need for this function. With dbi_ro_wr_{en/dis}, it's better to
> directly use the dbi accessors here.
I got it. I'll fix them.
> > +
> > + dw_pcie_dbi_ro_wr_dis(dw);
> > +
> > + if (IS_ENABLED(CONFIG_PCI_MSI)) {
> > + /* Enable MSI interrupt signal */
> > + val = readl(rcar->base + PCIEINTSTS0EN);
> > + val |= MSI_CTRL_INT;
> > + writel(val, rcar->base + PCIEINTSTS0EN);
> > + }
> > +
> > + gpiod_set_value_cansleep(dw->pe_rst, 0);
>
> If you end up adding perst assert above, add a 100ms delay before deassert.
I got it.
> > +
> > + return 0;
> > +}
> > +
>
> [...]
>
> > +
> > +static struct platform_driver rcar_gen4_pcie_driver = {
> > + .driver = {
> > + .name = "pcie-rcar-gen4",
> > + .of_match_table = rcar_gen4_pcie_of_match,
> > + },
> > + .probe = rcar_gen4_pcie_probe,
> > + .remove = rcar_gen4_pcie_remove,
>
> You should consider adding PROBE_PREFER_ASYNCHRONOUS here to avoid blocking
> other drivers while waiting for link_up during boot.
I see. This controller can add PROBE_PREFER_ASYNCHRONOUS without any problem.
So, I'll add it on v13.
> > +};
> > +module_platform_driver(rcar_gen4_pcie_driver);
> > +
> > +MODULE_DESCRIPTION("Renesas R-Car Gen4 PCIe host controller driver");
> > +MODULE_LICENSE("GPL");
> > diff --git a/drivers/pci/controller/dwc/pcie-rcar-gen4.c b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > new file mode 100644
> > index 000000000000..89cec76a41ab
> > --- /dev/null
> > +++ b/drivers/pci/controller/dwc/pcie-rcar-gen4.c
> > @@ -0,0 +1,187 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * PCIe host/endpoint controller driver for Renesas R-Car Gen4 Series SoCs
> > + * Copyright (C) 2022-2023 Renesas Electronics Corporation
> > + */
> > +
> > +#include <linux/delay.h>
> > +#include <linux/io.h>
> > +#include <linux/of_device.h>
> > +#include <linux/pci.h>
> > +#include <linux/pm_runtime.h>
> > +#include <linux/reset.h>
> > +
> > +#include "pcie-rcar-gen4.h"
> > +#include "pcie-designware.h"
> > +
> > +/* Renesas-specific */
> > +#define PCIERSTCTRL1 0x0014
> > +#define APP_HOLD_PHY_RST BIT(16)
> > +#define APP_LTSSM_ENABLE BIT(0)
> > +
> > +#define RETRAIN_MAX_RETRY 10
> > +
> > +static void rcar_gen4_pcie_ltssm_enable(struct rcar_gen4_pcie *rcar,
> > + bool enable)
> > +{
> > + u32 val;
> > +
> > + val = readl(rcar->base + PCIERSTCTRL1);
> > + if (enable) {
> > + val |= APP_LTSSM_ENABLE;
> > + val &= ~APP_HOLD_PHY_RST;
> > + } else {
> > + val &= ~APP_LTSSM_ENABLE;
> > + val |= APP_HOLD_PHY_RST;
> > + }
> > + writel(val, rcar->base + PCIERSTCTRL1);
> > +}
> > +
> > +static bool rcar_gen4_pcie_check_retrain_link(struct dw_pcie *dw)
> > +{
> > + u8 offset = dw_pcie_find_capability(dw, PCI_CAP_ID_EXP);
> > + u32 lnkcap = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCAP);
> > + u32 lnkctl = dw_pcie_readl_dbi(dw, offset + PCI_EXP_LNKCTL);
> > + u16 lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > + int i;
> > +
> > + if ((lnksta & PCI_EXP_LNKSTA_CLS) == (lnkcap & PCI_EXP_LNKCAP_SLS))
> > + return true;
> > +
> > + lnkctl |= PCI_EXP_LNKCTL_RL;
> > + dw_pcie_writel_dbi(dw, offset + PCI_EXP_LNKCTL, lnkctl);
> > +
> > + for (i = 0; i < RETRAIN_MAX_RETRY; i++) {
> > + lnksta = dw_pcie_readw_dbi(dw, offset + PCI_EXP_LNKSTA);
> > + if (lnksta & PCI_EXP_LNKSTA_LT)
> > + return true;
> > + usleep_range(1000, 1100);
> > + }
> > +
> > + return false;
> > +}
> > +
> > +static int rcar_gen4_pcie_link_up(struct dw_pcie *dw)
> > +{
> > + struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > + u32 val, mask;
> > +
> > + /* Require retraining here. Otherwise RDLH_LINK_UP may not be set */
>
> Any other platform expected to not require retrain?
I'm afraid but I don't know other platforms' requirement... But,
the RDLH_LINK_UP of PCIEINTSTS0 register is this controller specific,
so that retrain is required here.
> > + if (rcar->needs_retrain && !rcar_gen4_pcie_check_retrain_link(dw))
> > + return 0;
> > +
> > + val = readl(rcar->base + PCIEINTSTS0);
> > + mask = RDLH_LINK_UP | SMLH_LINK_UP;
> > +
> > + return (val & mask) == mask;
> > +}
> > +
> > +static int rcar_gen4_pcie_start_link(struct dw_pcie *dw)
> > +{
> > + struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +
> > + rcar_gen4_pcie_ltssm_enable(rcar, true);
> > +
> > + return 0;
> > +}
> > +
> > +static void rcar_gen4_pcie_stop_link(struct dw_pcie *dw)
> > +{
> > + struct rcar_gen4_pcie *rcar = to_rcar_gen4_pcie(dw);
> > +
> > + rcar_gen4_pcie_ltssm_enable(rcar, false);
> > +}
> > +
> > +int rcar_gen4_pcie_set_device_type(struct rcar_gen4_pcie *rcar, bool rc,
> > + int num_lanes)
> > +{
> > + u32 val;
> > +
> > + /* Note: Assume the reset is asserted here */
>
> What about this assumption?
Oops. "the reset" means that this controller's one in the SoC which
is controlled by a reset_control driver.. I should have revised this
description after I added pe_rst handling. I'll revise this comment on v13.
> > + val = readl(rcar->base + PCIEMSR0);
> > + if (rc)
> > + val |= DEVICE_TYPE_RC;
> > + else
> > + val |= DEVICE_TYPE_EP;
>
> newline
I got it.
> > + if (num_lanes < 4)
> > + val |= BIFUR_MOD_SET_ON;
>
> newline
I got it.
> > + writel(val, rcar->base + PCIEMSR0);
> > +
> > + return reset_control_deassert(rcar->rst);
> > +}
> > +
> > +void rcar_gen4_pcie_disable_bar(struct dw_pcie *dw, u32 bar_mask_reg)
> > +{
> > + dw_pcie_writel_dbi2(dw, bar_mask_reg, 0x0);
> > +}
> > +
> > +int rcar_gen4_pcie_prepare(struct rcar_gen4_pcie *rcar)
> > +{
> > + struct device *dev = rcar->dw.dev;
> > + int err;
> > +
> > + pm_runtime_enable(dev);
> > + err = pm_runtime_resume_and_get(dev);
>
> Why do you need do runtime_resume here? You don't have the runtime PM callbacks
> implemented. Even if you did, it may end up with a crash as you have't called
> dw_pcie_host_init().
The runtime PM APIs can handle power-domain and common clock framework.
Especially, I intended to handle common clock framework by runtime PM without
any runtime PM callbacks. In other words, runtime_resume is required before
dw_pcie_host_init() is called. Otherwise, any registers of this PCIe controller
cannot be accessed.
> Overall, I think you don't need to call any of the pm_runtime APIs now.
>
> > + if (err < 0) {
> > + dev_err(dev, "Failed to resume/get Runtime PM\n");
> > + pm_runtime_disable(dev);
> > + }
> > +
> > + return err;
> > +}
> > +
> > +void rcar_gen4_pcie_unprepare(struct rcar_gen4_pcie *rcar)
> > +{
> > + struct device *dev = rcar->dw.dev;
> > +
> > + if (!reset_control_status(rcar->rst))
> > + reset_control_assert(rcar->rst);
> > + pm_runtime_put(dev);
> > + pm_runtime_disable(dev);
> > +}
> > +
> > +static int rcar_gen4_pcie_devm_reset_get(struct rcar_gen4_pcie *rcar,
> > + struct device *dev)
> > +{
> > + rcar->rst = devm_reset_control_get(dev, NULL);
> > + if (IS_ERR(rcar->rst)) {
> > + dev_err(dev, "Failed to get Cold-reset\n");
>
> Is this cold-reset or core-reset?
This is cold-reset.
Best regards,
Yoshihiro Shimoda
> - Mani
