Hello Sebastian,
On Fri, Oct 20, 2023 at 11:23:57PM +0200, Sebastian Reichel wrote:
> Hello Niklas,
>
> On Fri, Oct 20, 2023 at 02:52:20PM +0200, Niklas Cassel wrote:
> > From: Niklas Cassel <niklas.cassel@xxxxxxx>
> >
> > From the snps,dw-pcie.yaml devicetree binding:
> > "At least DBI reg-space and peripheral devices CFG-space outbound window
> > are required for the normal controller work. iATU memory IO region is
> > also required if the space is unrolled (IP-core version >= 4.80a)."
> >
> > All the PCIe controllers in rk3588 are using the iATU unroll feature,
> > and thus have to supply the atu property in the device tree node.
> >
> > The size of the iATU region equals to:
> > MAX(num inbound ATU regions, num outbound ATU regions) * 0x200.
> >
> > Where for each 0x200 region, the registers controlling the
> > IATU_REGION_OUTBOUND starts at offset 0x0, and the registers controlling
> > IATU_REGION_INBOUND starts at offset 0x100.
> >
> > pcie3x4 and pcie3x2 have 16 ATU inbound regions, 16 ATU outbound regions,
> > so they have size: max(16, 16) * 0x200 = 0x2000
> >
> > pcie2x1l0, pcie2x1l1, and pcie2x1l2 have 8 ATU inbound regions, 8 ATU
> > outbound regions, so they have size: max(8, 8) * 0x200 = 0x1000
> >
> > On the rk3588 based rock-5b board:
> > Before this patch, dw_pcie_iatu_detect() fails to detect all iATUs:
> > rockchip-dw-pcie a40000000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
> > rockchip-dw-pcie a41000000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
> > rockchip-dw-pcie a40800000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
> >
> > After this patch, dw_pcie_iatu_detect() succeeds to detect all iATUs:
> > rockchip-dw-pcie a40000000.pcie: iATU: unroll T, 16 ob, 16 ib, align 64K, limit 8G
> > rockchip-dw-pcie a41000000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
> > rockchip-dw-pcie a40800000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
> >
> > Fixes: 8d81b77f4c49 ("arm64: dts: rockchip: add rk3588 PCIe2 support")
> > Fixes: 0acf4fa7f187 ("arm64: dts: rockchip: add PCIe3 support for rk3588")
> > Signed-off-by: Niklas Cassel <niklas.cassel@xxxxxxx>
> > ---
>
> Thanks for your patch. This looks sensible, but I have two comments:
>
> 1. You need to update the Rockchip DT binding. It currently enforces that regnames
> must be "dbi", "apb", "config". Thus 'make CHECK_DTBS=y rockchip/rk3588-rock-5b.dtb'
> will introduce new errors after this patch is applied.
In the rockchip-dw-pcie.yaml:
RK3568 SoC PCIe host controller is based on the Synopsys DesignWare
PCIe IP and thus inherits all the common properties defined in
snps,dw-pcie.yaml.
allOf:
- $ref: /schemas/pci/snps,dw-pcie.yaml#
So it obviously references the snps,dw-pcie.yaml using allOf. snps,dw-pcie.yaml
is the schema that has the atu property defined. If the validation tooling does
not support inherited properties, that is a bit disappointing, but not something
that I think should stop this patch, which solves a real problem, from being
accepted.
>
> 2. You calculated the exact ATU size and used that, but the binding specifies,
> "iATU/eDMA registers common for all device functions. [...] Note iATU is normally
> mapped to the 0x0 address of this region, while eDMA is available at 0x80000 base
> address.". So the size should be big enough to include the DMA section. So I would
> expect it to be 0x100000 for all controllers.
The device tree schema also allows you to specify the eDMA region using a separate
"dma" property.
This is what I chose to do, see my patch in:
https://lore.kernel.org/linux-devicetree/20231020224412.3722784-1-nks@xxxxxxxxxxx/T/#u
There are many reasons for this:
1) Having both "iatu" and "dma" in the same property is just plain confusing...
There are two separate properties for this in the DT schema, so it feels way
clearer to specify them explicitly, especially since they are not contiguous.
(There are a bunch of registers that are currently unused in the register space
between iATU and eDMA registers. How do we know that Synopsys does not decide
to make use of these unused registers for something else in a newer version of
the IP core?)
2) The rk3588 SoC has 5 PCIe controllers, but only one of them supports eDMA.
It feels better to specify the actual size of the iatu region for all, and
then add the dma region for the 1 controller that actually supports it,
rather than to specify an iatu region that is larger than it actually is for
the other 4 controllers.
3) This matches the way the existing device trees are defined:
$ git grep -B 10 \"atu\" arch/arm64/boot/dts/
I cannot see any device tree that has defined a "atu" region larger than
0x4000. (Which means that none of them are large enough to encompass the
dma region.)
And we know that at least some of these controllers support eDMA:
https://lore.kernel.org/linux-pci/20230717065459.14138-4-manivannan.sadhasivam@xxxxxxxxxx/
4) Even if I did specify a really large "atu" region, which is larger
than the actual atu region, and which is large enough to encompass the
dma region, the eDMA would still not work.
In order for the eDMA to work, it also requires the proper irqs defined,
specified using interrupt-names = "dmaX"... which makes most sense to combine
with the "dma" property in regs. (Looking at dmaX irq and then reaching the
conclusion that eDMA is actually part of the "atu" range is not really
intuitive.)
All in all, I think it is way clearer to specify them separately.
Or the driver should have went the other way, to not need to specify
"atu" at all, since both "atu" and "dma" are by default part of "dbi",
so the driver should theoretically be able to use default values for
these if the "dbi" range is defined sufficiently large...
but that is not how the DT schema or how the driver is currently designed.
(Since the DT schema says that you MUST specify "atu" if IP-core version
>= 4.80a.)
Kind regards,
Niklas
>
> Greetings,
>
> -- Sebastian
>
> > arch/arm64/boot/dts/rockchip/rk3588.dtsi | 21 ++++++++++++---------
> > arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 14 ++++++++------
> > 2 files changed, 20 insertions(+), 15 deletions(-)
> >
> > diff --git a/arch/arm64/boot/dts/rockchip/rk3588.dtsi b/arch/arm64/boot/dts/rockchip/rk3588.dtsi
> > index 5519c1430cb7..d7998a9c0c43 100644
> > --- a/arch/arm64/boot/dts/rockchip/rk3588.dtsi
> > +++ b/arch/arm64/boot/dts/rockchip/rk3588.dtsi
> > @@ -119,10 +119,11 @@ pcie3x4: pcie@fe150000 {
> > ranges = <0x01000000 0x0 0xf0100000 0x0 0xf0100000 0x0 0x00100000>,
> > <0x02000000 0x0 0xf0200000 0x0 0xf0200000 0x0 0x00e00000>,
> > <0x03000000 0x0 0x40000000 0x9 0x00000000 0x0 0x40000000>;
> > - reg = <0xa 0x40000000 0x0 0x00400000>,
> > + reg = <0xa 0x40000000 0x0 0x00300000>,
> > <0x0 0xfe150000 0x0 0x00010000>,
> > - <0x0 0xf0000000 0x0 0x00100000>;
> > - reg-names = "dbi", "apb", "config";
> > + <0x0 0xf0000000 0x0 0x00100000>,
> > + <0xa 0x40300000 0x0 0x00002000>;
> > + reg-names = "dbi", "apb", "config", "atu";
> > resets = <&cru SRST_PCIE0_POWER_UP>, <&cru SRST_P_PCIE0>;
> > reset-names = "pwr", "pipe";
> > status = "disabled";
> > @@ -170,10 +171,11 @@ pcie3x2: pcie@fe160000 {
> > ranges = <0x01000000 0x0 0xf1100000 0x0 0xf1100000 0x0 0x00100000>,
> > <0x02000000 0x0 0xf1200000 0x0 0xf1200000 0x0 0x00e00000>,
> > <0x03000000 0x0 0x40000000 0x9 0x40000000 0x0 0x40000000>;
> > - reg = <0xa 0x40400000 0x0 0x00400000>,
> > + reg = <0xa 0x40400000 0x0 0x00300000>,
> > <0x0 0xfe160000 0x0 0x00010000>,
> > - <0x0 0xf1000000 0x0 0x00100000>;
> > - reg-names = "dbi", "apb", "config";
> > + <0x0 0xf1000000 0x0 0x00100000>,
> > + <0xa 0x40700000 0x0 0x00002000>;
> > + reg-names = "dbi", "apb", "config", "atu";
> > resets = <&cru SRST_PCIE1_POWER_UP>, <&cru SRST_P_PCIE1>;
> > reset-names = "pwr", "pipe";
> > status = "disabled";
> > @@ -219,10 +221,11 @@ pcie2x1l0: pcie@fe170000 {
> > ranges = <0x01000000 0x0 0xf2100000 0x0 0xf2100000 0x0 0x00100000>,
> > <0x02000000 0x0 0xf2200000 0x0 0xf2200000 0x0 0x00e00000>,
> > <0x03000000 0x0 0x40000000 0x9 0x80000000 0x0 0x40000000>;
> > - reg = <0xa 0x40800000 0x0 0x00400000>,
> > + reg = <0xa 0x40800000 0x0 0x00300000>,
> > <0x0 0xfe170000 0x0 0x00010000>,
> > - <0x0 0xf2000000 0x0 0x00100000>;
> > - reg-names = "dbi", "apb", "config";
> > + <0x0 0xf2000000 0x0 0x00100000>,
> > + <0xa 0x40b00000 0x0 0x00001000>;
> > + reg-names = "dbi", "apb", "config", "atu";
> > resets = <&cru SRST_PCIE2_POWER_UP>, <&cru SRST_P_PCIE2>;
> > reset-names = "pwr", "pipe";
> > #address-cells = <3>;
> > diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> > index 5544f66c6ff4..286d7b10b9dd 100644
> > --- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> > +++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
> > @@ -1259,10 +1259,11 @@ pcie2x1l1: pcie@fe180000 {
> > ranges = <0x01000000 0x0 0xf3100000 0x0 0xf3100000 0x0 0x00100000>,
> > <0x02000000 0x0 0xf3200000 0x0 0xf3200000 0x0 0x00e00000>,
> > <0x03000000 0x0 0x40000000 0x9 0xc0000000 0x0 0x40000000>;
> > - reg = <0xa 0x40c00000 0x0 0x00400000>,
> > + reg = <0xa 0x40c00000 0x0 0x00300000>,
> > <0x0 0xfe180000 0x0 0x00010000>,
> > - <0x0 0xf3000000 0x0 0x00100000>;
> > - reg-names = "dbi", "apb", "config";
> > + <0x0 0xf3000000 0x0 0x00100000>,
> > + <0xa 0x40f00000 0x0 0x00001000>;
> > + reg-names = "dbi", "apb", "config", "atu";
> > resets = <&cru SRST_PCIE3_POWER_UP>, <&cru SRST_P_PCIE3>;
> > reset-names = "pwr", "pipe";
> > #address-cells = <3>;
> > @@ -1310,10 +1311,11 @@ pcie2x1l2: pcie@fe190000 {
> > ranges = <0x01000000 0x0 0xf4100000 0x0 0xf4100000 0x0 0x00100000>,
> > <0x02000000 0x0 0xf4200000 0x0 0xf4200000 0x0 0x00e00000>,
> > <0x03000000 0x0 0x40000000 0xa 0x00000000 0x0 0x40000000>;
> > - reg = <0xa 0x41000000 0x0 0x00400000>,
> > + reg = <0xa 0x41000000 0x0 0x00300000>,
> > <0x0 0xfe190000 0x0 0x00010000>,
> > - <0x0 0xf4000000 0x0 0x00100000>;
> > - reg-names = "dbi", "apb", "config";
> > + <0x0 0xf4000000 0x0 0x00100000>,
> > + <0xa 0x41300000 0x0 0x00001000>;
> > + reg-names = "dbi", "apb", "config", "atu";
> > resets = <&cru SRST_PCIE4_POWER_UP>, <&cru SRST_P_PCIE4>;
> > reset-names = "pwr", "pipe";
> > #address-cells = <3>;
> > --
> > 2.41.0
> >
