Re: [PATCH RFC 0/5] arm64: rockchip: enable PCIE3 controller and its phy for Rock5B boards

[Qu Wenruo <wqu@xxxxxxxx>]

On 2023/3/9 20:02, Piotr Oniszczuk wrote:
> > Wiadomość napisana przez Peter Geis <pgwipeout@xxxxxxxxx 
> > <mailto:pgwipeout@xxxxxxxxx>> w dniu 22.02.2023, o godz. 02:25:
> >
> > On Tue, Feb 21, 2023 at 4:45 PM Sebastian Reichel
> > <sebastian.reichel@xxxxxxxxxxxxx 
> > <mailto:sebastian.reichel@xxxxxxxxxxxxx>> wrote:
> > > Hi everyone,
> > >
> > > On Tue, Feb 21, 2023 at 01:55:48PM -0500, Peter Geis wrote:
> > > > On Tue, Feb 21, 2023 at 1:04 PM Piotr Oniszczuk
> > > > <piotr.oniszczuk@xxxxxxxxx <mailto:piotr.oniszczuk@xxxxxxxxx>> wrote:
> > > > > > Wiadomość napisana przez Qu Wenruo <wqu@xxxxxxxx 
> > > > > > <mailto:wqu@xxxxxxxx>> w dniu 21.02.2023, o godz. 01:14:
> > > > > > On 2023/2/21 02:25, Piotr Oniszczuk wrote:
> > > > > > > > Wiadomość napisana przez Qu Wenruo <wqu@xxxxxxxx 
> > > > > > > > <mailto:wqu@xxxxxxxx> <mailto:wqu@xxxxxxxx 
> > > > > > > > <mailto:wqu@xxxxxxxx>>> w dniu 04.02.2023, o godz. 09:47:
> > > > > > > >
> > > > > > > > This series is based on the existing upstream work from Sebastian
> > > > > > > > Reichel:
> > > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588 <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588> <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588 <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588>>
> > > > > > > >
> > > > > > > > And I'm a completely newbie to arm64 world, thus if there is 
> > > > > > > > something
> > > > > > > > wrong, feel free to point out and I'm pretty happy to learn from the
> > > > > > > > failure.
> > > > > > > >
> > > > > > > > [BACKGROUND]
> > > > > > > > RK3588S and RK3588 have PCIE supports, it's done by the following 3
> > > > > > > > controllers:
> > > > > > > >
> > > > > > > > - PCIE2.0x1 lane @fe180000 (both RK3588S and RK3588)
> > > > > > > > - PCIE2.0x1 lane @fe190000 (both RK3588S and RK3588)
> > > > > > > > Thes two are all connected to a naneng combo phy each, normally 
> > > > > > > > shared
> > > > > > > > with SATA or USB.
> > > > > > > >
> > > > > > > > - PCIE2.0x1 lane @fe170000 (RK3588 exlusive)
> > > > > > > > This one is also connected to a naneng combo phy, normally shared
> > > > > > > > with SATA or USB.
> > > > > > > >
> > > > > > > > - PCIE3.0x4 lanes @fe15000 (RK3588 exclusive)
> > > > > > > >
> > > > > > > > And unlike other boards, ROCK5B is utilizing PCIE extensively, its
> > > > > > > > network controller (RTL8125 2.5Gbps Ethernet) is connected to the
> > > > > > > > PCIE2.0 lane at fe190000, and an M.2 slot is attached to the 
> > > > > > > > PCIE3.0x4
> > > > > > > > lanes.
> > > > > > > >
> > > > > > > > [WORKING]
> > > > > > > > Currently the series is able to bring up the PCIE3.0x4 lanes and
> > > > > > > > properly boot from an NVME at that M.2 slot of Rock5B boards.
> > > > > > > >
> > > > > > > > [NOT WORKING]
> > > > > > > > All PCIE2.0 lanes connected to naneng combo phy are not working.
> > > > > > > > I tried forward porting the extra handling from downstream, but 
> > > > > > > > it only
> > > > > > > > results hanging at probing (causing RCU stall).
> > > > > > > >
> > > > > > > > [EXTRA WANRING]
> > > > > > > > - PCI MSI initialization warning
> > > > > > > > WARNING: CPU: 7 PID: 1 at drivers/pci/msi/msi.h:121 
> > > > > > > > pci_msi_setup_msi_irqs+0x38/0x4c
> > > > > > > >
> > > > > > > > This seems to be caused by the fact that we are still using legcacy
> > > > > > > > msi irqs?
> > > > > > > >
> > > > > > > > I checked up the gic and its dts, can not figure out why (all pretty
> > > > > > > > the same just like rk3399 and rk3568).
> > > > > > > > Any help would be appreciated.
> > > > > > > >
> > > > > > > > - pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under 
> > > > > > > > [bus 00-0f] (conflicts with (null) [bus 00-0f])
> > > > > > > > The vendoer kernel also has this problem, but my RK3399 board with
> > > > > > > > upstream kernel didn't trigger this at all, but something else like:
> > > > > > > >
> > > > > > > >   pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), 
> > > > > > > > reconfiguring
> > > > > > > >
> > > > > > > > Then:
> > > > > > > >
> > > > > > > >   pci_bus 0000:01: busn_res: [bus 01-1f] end is updated to 01
> > > > > > > >
> > > > > > > > Not sure if it's something missing or can be just ignored.
> > > > > > > >
> > > > > > > > [PCI DMESG]
> > > > > > > > With this patchset, the PCI initialization and nvme would look 
> > > > > > > > like this:
> > > > > > > >
> > > > > > > > [    0.142984] rockchip-dw-pcie fe150000.pcie: host bridge 
> > > > > > > > /pcie@fe150000 ranges:
> > > > > > > > [    0.143653] rockchip-dw-pcie fe150000.pcie:       IO 
> > > > > > > > 0x00f0100000..0x00f01fffff -> 0x00f0100000
> > > > > > > > [    0.144463] rockchip-dw-pcie fe150000.pcie:      MEM 
> > > > > > > > 0x00f0200000..0x00f0ffffff -> 0x00f0200000
> > > > > > > > [    0.145261] rockchip-dw-pcie fe150000.pcie:      MEM 
> > > > > > > > 0x0900000000..0x093fffffff -> 0x0900000000
> > > > > > > > [    0.154022] rockchip-dw-pcie fe150000.pcie: iATU: unroll T, 8 
> > > > > > > > ob, 8 ib, align 64K, limit 8G
> > > > > > > > [    0.360415] rockchip-dw-pcie fe150000.pcie: PCIe Gen.3 x4 link up
> > > > > > > > [    0.361099] rockchip-dw-pcie fe150000.pcie: PCI host bridge 
> > > > > > > > to bus 0000:00
> > > > > > > > [    0.361731] pci_bus 0000:00: root bus resource [bus 00-0f]
> > > > > > > > [    0.362236] pci_bus 0000:00: root bus resource [io 
> > > > > > > >  0x0000-0xfffff] (bus address [0xf0100000-0xf01fffff])
> > > > > > > > [    0.363113] pci_bus 0000:00: root bus resource [mem 
> > > > > > > > 0xf0200000-0xf0ffffff]
> > > > > > > > [    0.363744] pci_bus 0000:00: root bus resource [mem 
> > > > > > > > 0x900000000-0x93fffffff pref]
> > > > > > > > [    0.364450] pci 0000:00:00.0: [1d87:3588] type 01 class 0x060400
> > > > > > > > [    0.365005] pci 0000:00:00.0: reg 0x10: [mem 
> > > > > > > > 0x00000000-0x3fffffff]
> > > > > > > > [    0.365583] pci 0000:00:00.0: reg 0x14: [mem 
> > > > > > > > 0x00000000-0x3fffffff]
> > > > > > > > [    0.366159] pci 0000:00:00.0: reg 0x38: [mem 
> > > > > > > > 0x00000000-0x0000ffff pref]
> > > > > > > > [    0.366801] pci 0000:00:00.0: supports D1 D2
> > > > > > > > [    0.367193] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> > > > > > > > [    0.368647] pci_bus 0000:01: busn_res: can not insert [bus 
> > > > > > > > 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
> > > > > > > > [    0.369681] pci 0000:01:00.0: [1d97:5216] type 00 class 0x010802
> > > > > > > > [    0.370277] pci 0000:01:00.0: reg 0x10: [mem 
> > > > > > > > 0x00000000-0x0000ffff 64bit]
> > > > > > > > [    0.370975] pci 0000:01:00.0: reg 0x30: [mem 
> > > > > > > > 0x00000000-0x0001ffff pref]
> > > > > > > > [    0.372130] pci 0000:00:00.0: BAR 0: no space for [mem size 
> > > > > > > > 0x40000000]
> > > > > > > > [    0.372742] pci 0000:00:00.0: BAR 0: failed to assign [mem 
> > > > > > > > size 0x40000000]
> > > > > > > > [    0.373381] pci 0000:00:00.0: BAR 1: no space for [mem size 
> > > > > > > > 0x40000000]
> > > > > > > > [    0.373988] pci 0000:00:00.0: BAR 1: failed to assign [mem 
> > > > > > > > size 0x40000000]
> > > > > > > > [    0.374628] pci 0000:00:00.0: BAR 14: assigned [mem 
> > > > > > > > 0xf0200000-0xf02fffff]
> > > > > > > > [    0.375259] pci 0000:00:00.0: BAR 6: assigned [mem 
> > > > > > > > 0xf0300000-0xf030ffff pref]
> > > > > > > > [    0.375923] pci 0000:01:00.0: BAR 6: assigned [mem 
> > > > > > > > 0xf0200000-0xf021ffff pref]
> > > > > > > > [    0.376590] pci 0000:01:00.0: BAR 0: assigned [mem 
> > > > > > > > 0xf0220000-0xf022ffff 64bit]
> > > > > > > > [    0.377281] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> > > > > > > > [    0.377762] pci 0000:00:00.0:   bridge window [mem 
> > > > > > > > 0xf0200000-0xf02fffff]
> > > > > > > > [    0.426841] pcieport 0000:00:00.0: PME: Signaling with IRQ 33
> > > > > > > > [    0.427487] pcieport 0000:00:00.0: AER: enabled with IRQ 33
> > > > > > > > [    0.625353] nvme nvme0: pci function 0000:01:00.0
> > > > > > > > [    0.625774] nvme 0000:01:00.0: enabling device (0000 -> 0002)
> > > > > > > > [    0.717069] nvme nvme0: Shutdown timeout set to 8 seconds
> > > > > > > > [    0.723025] nvme nvme0: allocated 64 MiB host memory buffer.
> > > > > > > > [    0.816820] nvme nvme0: 1/0/0 default/read/poll queues
> > > > > > > > [    0.818079] sdhci-pltfm: SDHCI platform and OF driver helper
> > > > > > > > [    0.820678]  nvme0n1: p1 p2
> > > > > > > Qu, all
> > > > > > > I’m playing with your work on my rock5b as I want to have working 
> > > > > > > Eth on rock5b.
> > > > > > > My code is from 
> > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588 <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588> <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588 <https://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-misc.git/log/?h=rk3588>> + your’s PCIE3 patches.
> > > > > > > SBC boots from sd card, I see PCIE related logs in dmesg. but no 
> > > > > > > rtl8125 is detected.
> > > > > > > PCIE logs are like this:
> > > > > > >    8.207810] rockchip-dw-pcie fe150000.pcie: host bridge 
> > > > > > > /pcie@fe150000 ranges:
> > > > > > > [    8.208501] rockchip-dw-pcie fe150000.pcie: Parsing ranges 
> > > > > > > property...
> > > > > > > [    8.209089] rockchip-dw-pcie fe150000.pcie:       IO 
> > > > > > > 0x00f0100000..0x00f01fffff -> 0x00f0100000
> > > > > > > [    8.209944] rockchip-dw-pcie fe150000.pcie:      MEM 
> > > > > > > 0x00f0200000..0x00f0ffffff -> 0x00f0200000
> > > > > > > [    8.210740] rockchip-dw-pcie fe150000.pcie:      MEM 
> > > > > > > 0x0900000000..0x093fffffff -> 0x0900000000
> > > > > > > [    8.218918] rockchip-dw-pcie fe150000.pcie: iATU: unroll T, 8 
> > > > > > > ob, 8 ib, align 64K, limit 8G
> > > > > > > [    9.324473] rockchip-dw-pcie fe150000.pcie: Phy link never came up
> > > > > > > [    9.325186] rockchip-dw-pcie fe150000.pcie: PCI host bridge to 
> > > > > > > bus 0000:00
> > > > > > > [    9.325786] pci_bus 0000:00: root bus resource [bus 00-0f]
> > > > > > > [    9.326266] pci_bus 0000:00: root bus resource [io 
> > > > > > >  0x0000-0xfffff] (bus address [0xf0100000-0xf01fffff])
> > > > > > > [    9.327097] pci_bus 0000:00: root bus resource [mem 
> > > > > > > 0xf0200000-0xf0ffffff]
> > > > > > > [    9.327713] pci_bus 0000:00: root bus resource [mem 
> > > > > > > 0x900000000-0x93fffffff pref]
> > > > > > > [    9.328364] pci_bus 0000:00: scanning bus
> > > > > > > [    9.328729] pci 0000:00:00.0: [1d87:3588] type 01 class 0x060400
> > > > > > > [    9.329258] pci 0000:00:00.0: reg 0x10: [mem 
> > > > > > > 0x00000000-0x3fffffff]
> > > > > > > [    9.329807] pci 0000:00:00.0: reg 0x14: [mem 
> > > > > > > 0x00000000-0x3fffffff]
> > > > > > > [    9.330354] pci 0000:00:00.0: reg 0x38: [mem 
> > > > > > > 0x00000000-0x0000ffff pref]
> > > > > > > [    9.330984] pci 0000:00:00.0: supports D1 D2
> > > > > > > [    9.331356] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
> > > > > > > [    9.331858] pci 0000:00:00.0: PME# disabled
> > > > > > > [    9.332309] pci 0000:00:00.0: vgaarb: pci_notify
> > > > > > > [    9.333735] pci_bus 0000:00: fixups for bus
> > > > > > > [    9.334106] pci 0000:00:00.0: scanning [bus 01-ff] behind 
> > > > > > > bridge, pass 0
> > > > > > > [    9.334731] pci_bus 0000:01: busn_res: can not insert [bus 
> > > > > > > 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
> > > > > > > [    9.335668] pci_bus 0000:01: scanning bus
> > > > > > > [    9.336052] pci_bus 0000:01: fixups for bus
> > > > > > > [    9.336416] pci_bus 0000:01: bus scan returning with max=01
> > > > > > > [    9.336903] pci 0000:00:00.0: scanning [bus 01-ff] behind 
> > > > > > > bridge, pass 1
> > > > > > > [    9.337503] pci_bus 0000:00: bus scan returning with max=ff
> > > > > > > [    9.337994] pci 0000:00:00.0: BAR 0: no space for [mem size 
> > > > > > > 0x40000000]
> > > > > > > [    9.338570] pci 0000:00:00.0: BAR 0: failed to assign [mem 
> > > > > > > size 0x40000000]
> > > > > > > [    9.339175] pci 0000:00:00.0: BAR 1: no space for [mem size 
> > > > > > > 0x40000000]
> > > > > > > [    9.339749] pci 0000:00:00.0: BAR 1: failed to assign [mem 
> > > > > > > size 0x40000000]
> > > > > > > [    9.340356] pci 0000:00:00.0: BAR 6: assigned [mem 
> > > > > > > 0xf0200000-0xf020ffff pref]
> > > > > > > [    9.340991] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
> > > > > > > and nothing more :-(
> > > > > > > Are you progressing maybe with pcie on rock5b?
> > > > > > fe150000 is the PCIE3.0 controller, on Rock5B, that's the M.2 slot.
> > > > > >
> > > > > > But for R8125, it's on a PCIE2.0 controller, which is using naneng 
> > > > > > combo phy.
> > > > > >
> > > > > > I'm not able to bring the PCIE2.0 part up yet, it always hangs at 
> > > > > > PCIE2.0 initialization, thus only the PCIE3.0 part is submitted to 
> > > > > > the list.
> > > > > >
> > > > > > Thanks,
> > > > > > Qu
> > > > >
> > > > > Yes. Indeed.
> > > > > I'm trying to add pci2.0 and it looks i meet the same problem probably.
> > > > >
> > > > > I backported (from well working neggles quartz64pro repo I assume):
> > > > > https://github.com/neggles/linux-quartz64/commit/2ad84e89fc75b82c783345b72c97f5d7e3d45723 <https://github.com/neggles/linux-quartz64/commit/2ad84e89fc75b82c783345b72c97f5d7e3d45723>
> > > > > https://github.com/neggles/linux-quartz64/commit/4ac44c2b53758eca671d695d19b456d1849d7e14
> > > > > https://github.com/neggles/linux-quartz64/commit/714c5e01d8f0f73b3a49cbdee29e3ffe0f3353dd
> > > > > https://github.com/neggles/linux-quartz64/commit/64971f9c85f27e29c44b31a0c326ea4bb8ec3c56
> > > > >
> > > > > then i added rock5b pcie dt enablements (basing on radxa BSP):
> > > > > https://github.com/warpme/minimyth2/blob/master/script/kernel/linux-6.2/files/1058-arm64-dts-rockchip-enable-pcie-rock5b.patch
> > > > >
> > > > >
> > > > > this gives me quite clean 6.2 mainline boot log with hang at pcie2 
> > > > > init (pls see below)
> > > > >
> > > > > I assume 
> > > > > https://github.com/neggles/linux-quartz64/commits/linux-quartz64 
> > > > > repo works well on Quartsz64 board - so I indirectly assume above 
> > > > > backports should give us working pice2.0.
> > > > >
> > > > > It fails on rock5b - so I conclude: issue/error is on my side and 
> > > > > is related to my rock5b specifics.
> > > > > Unfortunately I don’t owe Quartz64 board so can't verify 
> > > > > correctness of my backports by testing on Quartz64pro.
> > > > >
> > > > > I’m curious about opinion of smarter than me guys…
> > > >
> > > > If the phy is misconfigured, not powered, or the clocks aren't going
> > > > active, you'll hang when the controller tries to touch it. Unless
> > > > someone has completed the combophy rk3588 bits the driver is not
> > > > functional yet for rk3588.
> > > >
> > > > Looking at the current 6.2 release, I only see the rk3568 compatible,
> > > > so you'll need to add support for rk3588 before it will work.
> > > >
> > > > Very Respectfully,
> > > > Peter Geis
> > >
> > > Sorry for being late to the game. Life is busy right now :)
> > >
> > > I haven't looked into PCIe myself so far, but some of my colleagues
> > > are looking into native network support on Rock 5B (and thus PCIe2
> > > controller).
> > >
> > > Apart from the obvious (missing rk3588 support in the combophy driver),
> > > the clocks will need some work. The clock tree implementation I 
> > > upstreamed
> > > is different from the downstream implementation. Downstream has some
> > > clocks that have two parent clocks using a hacked implementation
> > > that's obviously not upstreamable as is. The upstream implementation
> > > currently only describes the first parent. More details are in the
> > > following comment:
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/clk/rockchip/clk-rk3588.c#n16 <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/clk/rockchip/clk-rk3588.c#n16>
> > >
> > > Back than I wrote that I do not understand the exact need (the TRM
> > > does not describe the clock tree unfortuantely), I found this
> > > explanation:
> > >
> > > https://lore.kernel.org/dri-devel/20220309094139198367142@xxxxxxxxxxxxxx/
> > >
> > > To get the advanced blocks properly running upstream this needs a
> > > solution for this. Note that trying to access registers that are
> > > not clocked properly will result in a hang (as Peter already wrote).
> >
> > TLDR: There are now hardware blocks called a "Native Interface Unit"
> > (NIU) that gate the clocks to devices behind them. Essentially it
> > makes certain device clocks dependent on multiple parents being active
> > at the same time, which the current clock structure does not support.
> > It was decided that until the clock structure is updated to support
> > this, the NIU gate clocks would be marked critical and left always on.
> > Better to burn a miniscule more power than risk locking the board up.
> >
> > Though the issue here is much simpler than that, the controller simply
> > can't operate without a functional phy attached to it. They are
> > physically tied together and it's the controller < - > phy link that
> > doesn't function without the phy being configured. The combophy config
> > for rk3588 isn't significantly more complicated than rk356x, but one
> > can't mainline code that can't be tested and rk3588 wasn't available
> > at that time.
> >
> > > Apart from that the power-domain controller might need some of the
> > > extra bits downstream has.
> >
> > If PCIe 3 works, then this likely won't affect PCIe 2, though it
> > limits our ability to implement runtime power management.
> >
> > > Last but not least the GIC controller is handled differently in
> > > downstream. For that following the workaround that has been used for
> > > rk356x should also work for rk3588.
> >
> > The hack for the GIC controller that I maintain for rk356x is not
> > upstreamable. For the ITS to be supported in mainline we still require
> > an official errata from Rockchip documenting the issues here. The
> > mbi-alias method, while it works for simple configs, isn't scalable.
> >
> > > TLDR: This is not trivial. It's really unfortunate, that the board
> > > is not just using the native ethernet :(
> > >
> > > P.S.: We try to keep a rk3588 / Rock 5 status matrix maintained here:
> > >
> > > https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md <https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md>
> > >
> > > Greetings,
> > >
> > > — Sebastian
>
> Qu,
>
> Just fyi,
>
> I got Eth finally working on rock5b, so - if 6.2 kernel with mmc 
> bootable rock5b with ths, dvfs, mmc, pcie2, pcie3 and usb2 - will be in 
> any way helpful for you: patches over 6.2.2 mainline are here:
>
> https://github.com/warpme/minimyth2/tree/master/script/kernel/linux-6.2-with-rk3588/files <https://github.com/warpme/minimyth2/tree/master/script/kernel/linux-6.2-with-rk3588/files>
>
> pls look on patches: 1001-xx onwards + linux-6.2-arm64-armv8.config
>
> Code quality is far from upstreamable.
> Hope it will be convenient for any devel using rock5a/rock5b

Awesome! I can finally get rid of the stupid out-of-tree r8125 driver.

Would definitely have a good look into the patches and provide some 
feedbacks, just mind to share a git tree for easier reviewing?

Thanks,
Qu

> btw: if anybody is interested - i can provide Archlinux ARM sd card 
> bootstrap image for these boards…
> btw2: this huge list of patches is because i’m building single ArchLinux 
> ARM aarch64 kernel for 13+ SoCs in 17 different boards/tvboxes