Re: [PATCH RFC 0/5] arm64: rockchip: enable PCIE3 controller and its phy for Rock5B boards

[Qu Wenruo <wqu@xxxxxxxx>]

On 2023/2/21 02:25, Piotr Oniszczuk wrote:
> > Wiadomość napisana przez Qu Wenruo <wqu@xxxxxxxx 
> > <mailto:wqu@xxxxxxxx>> w dniu 04.02.2023, o godz. 09:47:
> >
> > This series is based on the existing upstream work from Sebastian
> > Reichel:
> > https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588 <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588>
> >
> > And I'm a completely newbie to arm64 world, thus if there is something
> > wrong, feel free to point out and I'm pretty happy to learn from the
> > failure.
> >
> > [BACKGROUND]
> > RK3588S and RK3588 have PCIE supports, it's done by the following 3
> > controllers:
> >
> > - PCIE2.0x1 lane @fe180000 (both RK3588S and RK3588)
> > - PCIE2.0x1 lane @fe190000 (both RK3588S and RK3588)
> >  Thes two are all connected to a naneng combo phy each, normally shared
> >  with SATA or USB.
> >
> > - PCIE2.0x1 lane @fe170000 (RK3588 exlusive)
> >  This one is also connected to a naneng combo phy, normally shared
> >  with SATA or USB.
> >
> > - PCIE3.0x4 lanes @fe15000 (RK3588 exclusive)
> >
> > And unlike other boards, ROCK5B is utilizing PCIE extensively, its
> > network controller (RTL8125 2.5Gbps Ethernet) is connected to the
> > PCIE2.0 lane at fe190000, and an M.2 slot is attached to the PCIE3.0x4
> > lanes.
> >
> > [WORKING]
> > Currently the series is able to bring up the PCIE3.0x4 lanes and
> > properly boot from an NVME at that M.2 slot of Rock5B boards.
> >
> > [NOT WORKING]
> > All PCIE2.0 lanes connected to naneng combo phy are not working.
> > I tried forward porting the extra handling from downstream, but it only
> > results hanging at probing (causing RCU stall).
> >
> > [EXTRA WANRING]
> > - PCI MSI initialization warning
> >  WARNING: CPU: 7 PID: 1 at drivers/pci/msi/msi.h:121 
> > pci_msi_setup_msi_irqs+0x38/0x4c
> >
> >  This seems to be caused by the fact that we are still using legcacy
> >  msi irqs?
> >
> >  I checked up the gic and its dts, can not figure out why (all pretty
> >  the same just like rk3399 and rk3568).
> >  Any help would be appreciated.
> >
> > - pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 
> > 00-0f] (conflicts with (null) [bus 00-0f])
> >  The vendoer kernel also has this problem, but my RK3399 board with
> >  upstream kernel didn't trigger this at all, but something else like:
> >
> >    pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), 
> > reconfiguring
> >
> >  Then:
> >
> >    pci_bus 0000:01: busn_res: [bus 01-1f] end is updated to 01
> >
> >  Not sure if it's something missing or can be just ignored.
> >
> > [PCI DMESG]
> > With this patchset, the PCI initialization and nvme would look like this:
> >
> > [    0.142984] rockchip-dw-pcie fe150000.pcie: host bridge 
> > /pcie@fe150000 ranges:
> > [    0.143653] rockchip-dw-pcie fe150000.pcie:       IO 
> > 0x00f0100000..0x00f01fffff -> 0x00f0100000
> > [    0.144463] rockchip-dw-pcie fe150000.pcie:      MEM 
> > 0x00f0200000..0x00f0ffffff -> 0x00f0200000
> > [    0.145261] rockchip-dw-pcie fe150000.pcie:      MEM 
> > 0x0900000000..0x093fffffff -> 0x0900000000
> > [    0.154022] rockchip-dw-pcie fe150000.pcie: iATU: unroll T, 8 ob, 8 
> > ib, align 64K, limit 8G
> > [    0.360415] rockchip-dw-pcie fe150000.pcie: PCIe Gen.3 x4 link up
> > [    0.361099] rockchip-dw-pcie fe150000.pcie: PCI host bridge to bus 
> > 0000:00
> > [    0.361731] pci_bus 0000:00: root bus resource [bus 00-0f]
> > [    0.362236] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] 
> > (bus address [0xf0100000-0xf01fffff])
> > [    0.363113] pci_bus 0000:00: root bus resource [mem 
> > 0xf0200000-0xf0ffffff]
> > [    0.363744] pci_bus 0000:00: root bus resource [mem 
> > 0x900000000-0x93fffffff pref]
> > [    0.364450] pci 0000:00:00.0: [1d87:3588] type 01 class 0x060400
> > [    0.365005] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
> > [    0.365583] pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
> > [    0.366159] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff 
> > pref]
> > [    0.366801] pci 0000:00:00.0: supports D1 D2
> > [    0.367193] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> > [    0.368647] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] 
> > under [bus 00-0f] (conflicts with (null) [bus 00-0f])
> > [    0.369681] pci 0000:01:00.0: [1d97:5216] type 00 class 0x010802
> > [    0.370277] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 
> > 64bit]
> > [    0.370975] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0001ffff 
> > pref]
> > [    0.372130] pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
> > [    0.372742] pci 0000:00:00.0: BAR 0: failed to assign [mem size 
> > 0x40000000]
> > [    0.373381] pci 0000:00:00.0: BAR 1: no space for [mem size 0x40000000]
> > [    0.373988] pci 0000:00:00.0: BAR 1: failed to assign [mem size 
> > 0x40000000]
> > [    0.374628] pci 0000:00:00.0: BAR 14: assigned [mem 
> > 0xf0200000-0xf02fffff]
> > [    0.375259] pci 0000:00:00.0: BAR 6: assigned [mem 
> > 0xf0300000-0xf030ffff pref]
> > [    0.375923] pci 0000:01:00.0: BAR 6: assigned [mem 
> > 0xf0200000-0xf021ffff pref]
> > [    0.376590] pci 0000:01:00.0: BAR 0: assigned [mem 
> > 0xf0220000-0xf022ffff 64bit]
> > [    0.377281] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> > [    0.377762] pci 0000:00:00.0:   bridge window [mem 
> > 0xf0200000-0xf02fffff]
> > [    0.426841] pcieport 0000:00:00.0: PME: Signaling with IRQ 33
> > [    0.427487] pcieport 0000:00:00.0: AER: enabled with IRQ 33
> > [    0.625353] nvme nvme0: pci function 0000:01:00.0
> > [    0.625774] nvme 0000:01:00.0: enabling device (0000 -> 0002)
> > [    0.717069] nvme nvme0: Shutdown timeout set to 8 seconds
> > [    0.723025] nvme nvme0: allocated 64 MiB host memory buffer.
> > [    0.816820] nvme nvme0: 1/0/0 default/read/poll queues
> > [    0.818079] sdhci-pltfm: SDHCI platform and OF driver helper
> > [    0.820678]  nvme0n1: p1 p2
>
> Qu, all
>
> I’m playing with your work on my rock5b as I want to have working Eth on 
> rock5b.
>
> My code is from 
> https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588 <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588> + your’s PCIE3 patches.
>
> SBC boots from sd card, I see PCIE related logs in dmesg. but no rtl8125 
> is detected.
> PCIE logs are like this:
>
>      8.207810] rockchip-dw-pcie fe150000.pcie: host bridge 
> /pcie@fe150000 ranges:
> [    8.208501] rockchip-dw-pcie fe150000.pcie: Parsing ranges property...
> [    8.209089] rockchip-dw-pcie fe150000.pcie:       IO 
> 0x00f0100000..0x00f01fffff -> 0x00f0100000
> [    8.209944] rockchip-dw-pcie fe150000.pcie:      MEM 
> 0x00f0200000..0x00f0ffffff -> 0x00f0200000
> [    8.210740] rockchip-dw-pcie fe150000.pcie:      MEM 
> 0x0900000000..0x093fffffff -> 0x0900000000
> [    8.218918] rockchip-dw-pcie fe150000.pcie: iATU: unroll T, 8 ob, 8 
> ib, align 64K, limit 8G
> [    9.324473] rockchip-dw-pcie fe150000.pcie: Phy link never came up
> [    9.325186] rockchip-dw-pcie fe150000.pcie: PCI host bridge to bus 
> 0000:00
> [    9.325786] pci_bus 0000:00: root bus resource [bus 00-0f]
> [    9.326266] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] 
> (bus address [0xf0100000-0xf01fffff])
> [    9.327097] pci_bus 0000:00: root bus resource [mem 
> 0xf0200000-0xf0ffffff]
> [    9.327713] pci_bus 0000:00: root bus resource [mem 
> 0x900000000-0x93fffffff pref]
> [    9.328364] pci_bus 0000:00: scanning bus
> [    9.328729] pci 0000:00:00.0: [1d87:3588] type 01 class 0x060400
> [    9.329258] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
> [    9.329807] pci 0000:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
> [    9.330354] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
> [    9.330984] pci 0000:00:00.0: supports D1 D2
> [    9.331356] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> [    9.331858] pci 0000:00:00.0: PME# disabled
> [    9.332309] pci 0000:00:00.0: vgaarb: pci_notify
> [    9.333735] pci_bus 0000:00: fixups for bus
> [    9.334106] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 0
> [    9.334731] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] 
> under [bus 00-0f] (conflicts with (null) [bus 00-0f])
> [    9.335668] pci_bus 0000:01: scanning bus
> [    9.336052] pci_bus 0000:01: fixups for bus
> [    9.336416] pci_bus 0000:01: bus scan returning with max=01
> [    9.336903] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 1
> [    9.337503] pci_bus 0000:00: bus scan returning with max=ff
> [    9.337994] pci 0000:00:00.0: BAR 0: no space for [mem size 0x40000000]
> [    9.338570] pci 0000:00:00.0: BAR 0: failed to assign [mem size 
> 0x40000000]
> [    9.339175] pci 0000:00:00.0: BAR 1: no space for [mem size 0x40000000]
> [    9.339749] pci 0000:00:00.0: BAR 1: failed to assign [mem size 
> 0x40000000]
> [    9.340356] pci 0000:00:00.0: BAR 6: assigned [mem 
> 0xf0200000-0xf020ffff pref]
> [    9.340991] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
>
> and nothing more :-(
>
> Are you progressing maybe with pcie on rock5b?
fe150000 is the PCIE3.0 controller, on Rock5B, that's the M.2 slot.

But for R8125, it's on a PCIE2.0 controller, which is using naneng combo 
phy.

I'm not able to bring the PCIE2.0 part up yet, it always hangs at 
PCIE2.0 initialization, thus only the PCIE3.0 part is submitted to the list.

Thanks,
Qu