On 26 September 2017 at 10:32, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> [+cc Will]
>
> On Mon, Aug 28, 2017 at 07:04:36PM +0100, Ard Biesheuvel wrote:
>> Some implementations of the Synopsys Designware PCIe controller implement
>> a so-called ECAM shift mode, which allows a static memory window to be
>> configured that covers the configuration space of the entire bus range.
>>
>> If the firmware performs all the low level configuration that is required
>> to expose this controller in a fully ECAM compatible manner, we can
>> simply describe it as "pci-host-ecam-generic" and be done with it.
>> However, it appears that in some cases (one of which is the Armada 80x0),
>> the IP is synthesized with an ATU window size that does not allow the
>> first bus to be mapped in a way that prevents the device on the
>> downstream port from appearing more than once.
>>
>> So implement a driver that relies on the firmware to perform all low
>> level initialization, and drives the controller in ECAM mode, but
>> overrides the config space accessors to take the above quirk into
>> account.
>>
>> Note that, unlike most drivers for this IP, this driver does not expose
>> a fake bridge device at B/D/F 00:00.0. There is no point in doing so,
>> given that this is not a true bridge, and does not require any windows
>> to be configured in order for the downstream device to operate correctly.
>> Omitting it also prevents the PCI resource allocation routines from
>> handing out BAR space to it unnecessarily.
>
> This is a tangent, but does this mean the other drivers do not need to
> expose a fake 00:00.0 device either?
>
To be honest, I am not so sure anymore. I am seeing some issues in
ASPM code making the assumption that any device which is not a root
port has a parent. If this is mandated by the spec, I guess there
isn't a whole lot we can do except expose a fake root port on b/d/f
0/0/0. This used to work fine, though, and I have to confirm whether
the issues I am seeing currently are due to different hardware or
changes in the software.
> s/Designware/DesignWare/ in comments, changelogs, Kconfig text, etc.
>
OK
>> Cc: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
>> Cc: Jingoo Han <jingoohan1@xxxxxxxxx>
>> Cc: Joao Pinto <Joao.Pinto@xxxxxxxxxxxx>
>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@xxxxxxxxxx>
>> ---
>> drivers/pci/dwc/Kconfig | 11 +++
>> drivers/pci/dwc/Makefile | 1 +
>> drivers/pci/dwc/pcie-designware-ecam.c | 77 ++++++++++++++++++++
>
> This really doesn't have any DesignWare specifics in it, and it seems
> more related to drivers/pci/host/pci-host-generic.c than to anything
> in drivers/pci/dwc. Maybe it should be
> drivers/pci/host/pci-host-generic-quirks.c or something? That's
> unwieldy, I admit.
>
I don't care where we put it, and I am fine with owning it if you prefer.
> Putting it in pci/dwc would make Jingoo and Joao the default
> maintainers; I don't know how they feel about that. We would probably
> have to tweak MAINTAINERS if we *didn't* put it in pci/dwc.
>
> Any thoughts on this, Will?
>
>> 3 files changed, 89 insertions(+)
>>
>> diff --git a/drivers/pci/dwc/Kconfig b/drivers/pci/dwc/Kconfig
>> index d275aadc47ee..477576d07911 100644
>> --- a/drivers/pci/dwc/Kconfig
>> +++ b/drivers/pci/dwc/Kconfig
>> @@ -169,4 +169,15 @@ config PCIE_KIRIN
>> Say Y here if you want PCIe controller support
>> on HiSilicon Kirin series SoCs.
>>
>> +config PCIE_DW_HOST_ECAM
>> + bool "Synopsys DesignWare PCIe controller in ECAM mode"
>> + depends on OF && PCI
>> + select PCI_HOST_COMMON
>> + select IRQ_DOMAIN
>> + help
>> + Add support for Synopsys DesignWare PCIe controllers configured
>> + by the firmware into ECAM shift mode. In some cases, these are
>> + fully ECAM compliant, in which case the pci-host-generic driver
>> + may be used instead.
>
> This doesn't quite read right. It sounds like a controller in ECAM
> shift mode might be fully ECAM compliant, but I don't think that's
> what you intended.
>
Yes, that is what I mean. ECAM shift mode results in a fully compliant
ECAM config space iff the IP was synthesized with a 32 KB granularity
for the iATU windows. The default is 64 KB, though, in which case you
need this driver.
> IIUC, the controller can be in either "ECAM shift mode" (where we need
> this new driver) or in a "fully ECAM compliant mode" (where we can use
> pci-host-generic).
>
No, this is not the case
>> endmenu
>> diff --git a/drivers/pci/dwc/Makefile b/drivers/pci/dwc/Makefile
>> index c61be9738cce..7d5a23e5b767 100644
>> --- a/drivers/pci/dwc/Makefile
>> +++ b/drivers/pci/dwc/Makefile
>> @@ -1,5 +1,6 @@
>> obj-$(CONFIG_PCIE_DW) += pcie-designware.o
>> obj-$(CONFIG_PCIE_DW_HOST) += pcie-designware-host.o
>> +obj-$(CONFIG_PCIE_DW_HOST_ECAM) += pcie-designware-ecam.o
>> obj-$(CONFIG_PCIE_DW_EP) += pcie-designware-ep.o
>> obj-$(CONFIG_PCIE_DW_PLAT) += pcie-designware-plat.o
>> ifneq ($(filter y,$(CONFIG_PCI_DRA7XX_HOST) $(CONFIG_PCI_DRA7XX_EP)),)
>> diff --git a/drivers/pci/dwc/pcie-designware-ecam.c b/drivers/pci/dwc/pcie-designware-ecam.c
>> new file mode 100644
>> index 000000000000..ede627d7d08b
>> --- /dev/null
>> +++ b/drivers/pci/dwc/pcie-designware-ecam.c
>> @@ -0,0 +1,77 @@
>> +/*
>> + * Driver for mostly ECAM compatible Synopsys dw PCIe controllers
>> + * configured by the firmware into RC mode
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * Copyright (C) 2014 ARM Limited
>> + * Copyright (C) 2017 Linaro Limited
>> + *
>> + * Authors: Will Deacon <will.deacon@xxxxxxx>
>> + * Ard Biesheuvel <ard.biesheuvel@xxxxxxxxxx>
>> + */
>> +
>> +#include <linux/kernel.h>
>> +#include <linux/init.h>
>> +#include <linux/of_address.h>
>> +#include <linux/of_pci.h>
>> +#include <linux/pci-ecam.h>
>> +#include <linux/platform_device.h>
>> +
>> +static int pci_dw_ecam_config_read(struct pci_bus *bus, u32 devfn, int where,
>> + int size, u32 *val)
>> +{
>> + struct pci_config_window *cfg = bus->sysdata;
>> +
>> + /*
>> + * The Synopsys dw PCIe controller in RC mode will not filter type 0
>> + * config TLPs sent to devices 1 and up on its downstream port,
>> + * resulting in devices appearing multiple times on bus 0 unless we
>> + * filter them here.
>> + */
>> + if (bus->number == cfg->busr.start && PCI_SLOT(devfn) > 0) {
>
> Trivial, but maybe you could factor out this test? We already have
> these functions that do basically the same thing and it'd be nice to
> use a similar pattern (altera and dw also check for the link being up,
> which seems racy and possibly bogus to me):
>
> altera_pcie_valid_device()
> dw_pcie_valid_device()
> rockchip_pcie_valid_device()
>
> The fact that altera and rockchip do essentially the same thing as dw
> here suggests that this pattern is not limited to DesignWare.
>
> These other functions also do something similar, though not structured
> the same way:
>
> hisi_pcie_rd_conf()
> advk_pcie_rd_conf()
> thunder_pem_bridge_read()
> rcar_pcie_config_access()
> gapspci_config_access()
>
I can look into that.
>> + *val = 0xffffffff;
>> + return PCIBIOS_DEVICE_NOT_FOUND;
>> + }
>> + return pci_generic_config_read(bus, devfn, where, size, val);
>> +}
>> +
>> +static int pci_dw_ecam_config_write(struct pci_bus *bus, u32 devfn, int where,
>> + int size, u32 val)
>> +{
>> + struct pci_config_window *cfg = bus->sysdata;
>> +
>> + if (bus->number == cfg->busr.start && PCI_SLOT(devfn) > 0)
>> + return PCIBIOS_DEVICE_NOT_FOUND;
>> +
>> + return pci_generic_config_write(bus, devfn, where, size, val);
>> +}
>> +
>> +static struct pci_ecam_ops pci_dw_ecam_bus_ops = {
>> + .pci_ops.map_bus = pci_ecam_map_bus,
>> + .pci_ops.read = pci_dw_ecam_config_read,
>> + .pci_ops.write = pci_dw_ecam_config_write,
>> + .bus_shift = 20,
>> +};
>> +
>> +static const struct of_device_id pci_dw_ecam_of_match[] = {
>> + { .compatible = "marvell,armada8k-pcie-ecam" },
>> + { .compatible = "socionext,synquacer-pcie-ecam" },
>> + { .compatible = "snps,dw-pcie-ecam" },
>> + { },
>> +};
>> +
>> +static int pci_dw_ecam_probe(struct platform_device *pdev)
>> +{
>> + return pci_host_common_probe(pdev, &pci_dw_ecam_bus_ops);
>> +}
>> +
>> +static struct platform_driver pci_dw_ecam_driver = {
>> + .driver.name = "pcie-designware-ecam",
>> + .driver.of_match_table = pci_dw_ecam_of_match,
>> + .driver.suppress_bind_attrs = true,
>> + .probe = pci_dw_ecam_probe,
>> +};
>> +builtin_platform_driver(pci_dw_ecam_driver);
>> --
>> 2.11.0
>>
