> -----Original Message-----
> From: Russell King <linux@xxxxxxxxxxxxxxx>
> Sent: 04 March 2025 10:00
> To: Biju Das <biju.das.jz@xxxxxxxxxxxxxx>
> Cc: Lad, Prabhakar <prabhakar.csengg@xxxxxxxxx>; Andrew Lunn <andrew+netdev@xxxxxxx>; David S. Miller
> <davem@xxxxxxxxxxxxx>; Eric Dumazet <edumazet@xxxxxxxxxx>; Jakub Kicinski <kuba@xxxxxxxxxx>; Paolo
> Abeni <pabeni@xxxxxxxxxx>; Rob Herring <robh@xxxxxxxxxx>; Krzysztof Kozlowski <krzk+dt@xxxxxxxxxx>;
> Conor Dooley <conor+dt@xxxxxxxxxx>; Philipp Zabel <p.zabel@xxxxxxxxxxxxxx>; Geert Uytterhoeven
> <geert+renesas@xxxxxxxxx>; Giuseppe Cavallaro <peppe.cavallaro@xxxxxx>; Jose Abreu
> <joabreu@xxxxxxxxxxxx>; Alexandre Torgue <alexandre.torgue@xxxxxxxxxxx>; netdev@xxxxxxxxxxxxxxx;
> devicetree@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; linux-renesas-soc@xxxxxxxxxxxxxxx; Fabrizio
> Castro <fabrizio.castro.jz@xxxxxxxxxxx>; Prabhakar Mahadev Lad <prabhakar.mahadev-
> lad.rj@xxxxxxxxxxxxxx>
> Subject: Re: [PATCH 3/3] net: stmmac: Add DWMAC glue layer for Renesas GBETH
>
> On Tue, Mar 04, 2025 at 06:58:44AM +0000, Biju Das wrote:
> > Hi Russel king,
> >
> > > -----Original Message-----
> > > From: Russell King <linux@xxxxxxxxxxxxxxx>
> > > Sent: 03 March 2025 11:19
> > > Subject: Re: [PATCH 3/3] net: stmmac: Add DWMAC glue layer for
> > > Renesas GBETH
> > >
> > > On Sun, Mar 02, 2025 at 10:02:15PM +0000, Lad, Prabhakar wrote:
> > > > Hi Russell,
> > > >
> > > > On Sun, Mar 2, 2025 at 9:44 PM Russell King (Oracle)
> > > > <linux@xxxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > On Sun, Mar 02, 2025 at 09:20:49PM +0000, Lad, Prabhakar wrote:
> > > > > > Hi Russell,
> > > > > > > What is the reason for setting this flag? If it's because of
> > > > > > > suspend/ resume failures, does my "net: stmmac: fix resume
> > > > > > > failures due to RX clock" series solve this for you without requiring this flag?
> > > > > > >
> > > > > > Ive set this flag based on the configuration supported by this IP.
> > > > > > Unfortunately the platform which I am working on doesn't
> > > > > > support s2r yet so I cannot test suspend/resume path yet. But
> > > > > > I do see an issue when I unload and load just the glue module the DMA reset fails.
> > > > >
> > > > > Thanks for that feedback - that's a scenario I hadn't considered.
> > > > >
> > > > > I was trying to avoid having to disable LPI RX clock-stop on
> > > > > suspend by ensuring that it was enabled at resume time. I think
> > > > > that's valid, but you've brought up another similar scenario:
> > > > >
> > > > > - device is brought up, configures RX clock stop
> > > > > - links with media, negotiates EEE
> > > > > - driver is unloaded, link doesn't go down, but due to no traffic goes
> > > > > into idle, so RX clock is stopped
> > > > > - driver reloaded, RX clock still stopped, reset fails
> > > > >
> > > > > I would like to solve that so we can get the power savings from
> > > > > stopping the clock, but still have reset work when necessary.
> > > > >
> > > > I would be happy to test the patches ;)
> > > >
> > > > > I'm guessing that the "DMA reset fails" refers to this path:
> > > > >
> > > > > stmmac_open() -> __stmmac_open() -> stmmac_hw_setup() ->
> > > > > stmmac_init_dma_engine() -> stmmac_reset() ?
> > > > >
> > > > Yes.
> > > >
> > > > > In other words, when the device is being brought back up
> > > > > adminsitratively?
> > > > >
> > > > > What happens if you (replace $if):
> > > > >
> > > > > # ip li set dev $if down
> > > > > # ip li set dev $if up
> > > > >
> > > > > Does that also fail without STMMAC_FLAG_RX_CLK_RUNS_IN_LPI set?
> > > > >
> > > > Logs without STMMAC_FLAG_RX_CLK_RUNS_IN_LPI flag set:
> > > > --------------------------------------------------------------
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth1 down
> > > > [ 33.606549] renesas-gbeth 15c40000.ethernet eth1: Link is Down
> > > > root@rzv2h-evk-alpha:~#
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth0 down
> > > > [ 37.356992] renesas-gbeth 15c30000.ethernet eth0: Link is Down
> > > > root@rzv2h-evk-alpha:~#
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth1 up
> > > > [ 43.974803] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-0
> > > > [ 43.983189] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-1
> > > > [ 43.991155] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-2
> > > > [ 43.999128] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-3
> > > > [ 44.072079] renesas-gbeth 15c40000.ethernet eth1: PHY [stmmac-1:00]
> > > > driver [Microchip KSZ9131 Gigabit PHY] (irq=POLL)
> > > > [ 44.094605] dwmac4: Master AXI performs fixed burst length
> > > > [ 44.100138] renesas-gbeth 15c40000.ethernet eth1: No Safety
> > > > Features support found
> > > > [ 44.107748] renesas-gbeth 15c40000.ethernet eth1: IEEE 1588-2008
> > > > Advanced Timestamp supported
> > > > [ 44.116725] renesas-gbeth 15c40000.ethernet eth1: registered PTP clock
> > > > [ 44.123352] renesas-gbeth 15c40000.ethernet eth1: configuring for
> > > > phy/rgmii-id link mode
> > > > root@rzv2h-evk-alpha:~#
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth1[ 47.207761] renesas-gbeth
> > > > 15c40000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
> > > > ^C root@rzv2h-evk-alpha:~# ^C root@rzv2h-evk-alpha:~# ip li set
> > > > dev eth0 up
> > > > [ 55.636722] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-0
> > > > [ 55.645139] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-1
> > > > [ 55.653111] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-2
> > > > [ 55.661073] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-3
> > > > [ 55.732087] renesas-gbeth 15c30000.ethernet eth0: PHY [stmmac-0:00]
> > > > driver [Microchip KSZ9131 Gigabit PHY] (irq=POLL)
> > > > [ 55.754612] dwmac4: Master AXI performs fixed burst length
> > > > [ 55.760143] renesas-gbeth 15c30000.ethernet eth0: No Safety
> > > > Features support found
> > > > [ 55.767740] renesas-gbeth 15c30000.ethernet eth0: IEEE 1588-2008
> > > > Advanced Timestamp supported
> > > > [ 55.776705] renesas-gbeth 15c30000.ethernet eth0: registered PTP clock
> > > > [ 55.783333] renesas-gbeth 15c30000.ethernet eth0: configuring for
> > > > phy/rgmii-id link mode
> > > > root@rzv2h-evk-alpha:~#
> > > > root@rzv2h-evk-alpha:~# [ 58.855844] renesas-gbeth 15c30000.ethernet
> > > > eth0: tx_clk_stop=1
> > > > [ 58.861989] renesas-gbeth 15c30000.ethernet eth0: Link is Up -
> > > > 1Gbps/Full - flow control rx/tx
> > > >
> > > > root@rzv2h-evk-alpha:~#
> > > > root@rzv2h-evk-alpha:~#
> > > >
> > > > Logs with STMMAC_FLAG_RX_CLK_RUNS_IN_LPI flag set:
> > > > --------------------------------------------------------------
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth1 down
> > > > [ 30.053790] renesas-gbeth 15c40000.ethernet eth1: Link is Down
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth0 down
> > > > [ 35.366935] renesas-gbeth 15c30000.ethernet eth0: Link is Down
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth1 up
> > > > [ 40.448563] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-0
> > > > [ 40.456725] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-1
> > > > [ 40.464893] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-2
> > > > [ 40.472840] renesas-gbeth 15c40000.ethernet eth1: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-3
> > > > [ 40.543895] renesas-gbeth 15c40000.ethernet eth1: PHY [stmmac-1:00]
> > > > driver [Microchip KSZ9131 Gigabit PHY] (irq=POLL)
> > > > [ 40.566419] dwmac4: Master AXI performs fixed burst length
> > > > [ 40.571949] renesas-gbeth 15c40000.ethernet eth1: No Safety
> > > > Features support found
> > > > [ 40.579550] renesas-gbeth 15c40000.ethernet eth1: IEEE 1588-2008
> > > > Advanced Timestamp supported
> > > > [ 40.588505] renesas-gbeth 15c40000.ethernet eth1: registered PTP clock
> > > > [ 40.595135] renesas-gbeth 15c40000.ethernet eth1: configuring for
> > > > phy/rgmii-id link mode
> > > > root@rzv2h-evk-alpha:~#
> > > > root@rzv2h-evk-alpha:~# [ 43.687551] renesas-gbeth 15c40000.ethernet
> > > > eth1: Link is Up - 1Gbps/Full - flow control off
> > > >
> > > > root@rzv2h-evk-alpha:~# ip li set dev eth0 up
> > > > [ 49.644479] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-0
> > > > [ 49.652719] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-1
> > > > [ 49.660681] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-2
> > > > [ 49.669059] renesas-gbeth 15c30000.ethernet eth0: Register
> > > > MEM_TYPE_PAGE_POOL RxQ-3
> > > > [ 49.740011] renesas-gbeth 15c30000.ethernet eth0: PHY [stmmac-0:00]
> > > > driver [Microchip KSZ9131 Gigabit PHY] (irq=POLL)
> > > > [ 49.762518] dwmac4: Master AXI performs fixed burst length
> > > > [ 49.768057] renesas-gbeth 15c30000.ethernet eth0: No Safety
> > > > Features support found
> > > > [ 49.775655] renesas-gbeth 15c30000.ethernet eth0: IEEE 1588-2008
> > > > Advanced Timestamp supported
> > > > [ 49.784609] renesas-gbeth 15c30000.ethernet eth0: registered PTP clock
> > > > [ 49.791236] renesas-gbeth 15c30000.ethernet eth0: configuring for
> > > > phy/rgmii-id link mode
> > > > root@rzv2h-evk-alpha:~#
> > > > root@rzv2h-evk-alpha:~# [ 52.871635] renesas-gbeth 15c30000.ethernet
> > > > eth0: tx_clk_stop=1
> > > > [ 52.877777] renesas-gbeth 15c30000.ethernet eth0: Link is Up -
> > > > 1Gbps/Full - flow control rx/tx
> > >
> > down/upping the interface but it seems you get different behaviour.
> > >
> > > I'd like to understand why that is, because at the moment I'm
> > > wondering whether my patches that address the suspend/resume need
> > > further work before I send them - but in order to assess that, I
> > > need to work out why your issue only seems to occur in the module removal/insertion and not
> down/up as well as I'd expect.
> >
> > FYI, With linux next, On RZ/G3E SoC which has similar IP as RZ/V2H,i
> > ethernet works during suspend entry/exit Even though STR is not fully
> > functional yet.
>
> For the failure to happen, you need to check whether EEE is being used:
>
> # ethtool --show-eee ethX
>
> and check whether it states that EEE is enabled and active, and Tx LPI also shows the timer value.
>
> You need a PHY that does stop it's receive clock when the link enters low-power mode. PHYs are not
> required to have this ability implemented, and there's no way for software to know whether it is or
> not.
>
> Then, you need to be certain that your link partner does actually support EEE and signals LPI from its
> side, rather than just advertising EEE. Lastly, you need to ensure that there is no traffic over the
> cable when you're resuming for the period of the reset timeout for the failure to occur. If the link
> wakes up, the clock will be started and reset will complete.
>
> One can rule out some of the above by checking the LPI status bits, either in the DWMAC or PHY which
> indicates whether transmit and/or receive seeing LPI signalled.
>
> If the link doesn't enter low power, then the receive clock won't be stopped, and reset will complete.
> If the link wakes up during reset, then the clock will be restarted, and reset will complete before
> the timeout expires.
>
> So, the possibility for a successful test is quite high.
This what I get on next. It is showing enabled , but inactive.
root@smarc-rzg3e:~# ethtool --show-eee eth0
EEE settings for eth0:
EEE status: enabled - inactive
Tx LPI: 1000000 (us)
Supported EEE link modes: 100baseT/Full
1000baseT/Full
Advertised EEE link modes: 100baseT/Full
1000baseT/Full
Link partner advertised EEE link modes: Not reported
root@smarc-rzg3e:~# [ 4255.568190] PM: suspend entry (s2idle)
[ 4255.572235] Filesystems sync: 0.000 seconds
[ 4255.577786] Freezing user space processes
[ 4255.585187] Freezing user space processes completed (elapsed 0.003 seconds)
[ 4255.592143] OOM killer disabled.
[ 4255.595375] Freezing remaining freezable tasks
[ 4255.600907] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
[ 4255.608297] printk: Suspending console(s) (use no_console_suspend to debug)
ÿ[ 4255.657045] renesas-gbeth 15c30000.ethernet eth0: Link is Down
...
...
[ 4255.659896] PM: Some devices failed to suspend, or early wake event detected
[ 4255.661252] renesas-gbeth 15c30000.ethernet eth0: configuring for phy/rgmii-id link mode
[ 4255.665681] dwmac4: Master AXI performs fixed burst length
[ 4255.665763] renesas-gbeth 15c30000.ethernet eth0: No Safety Features support found
[ 4255.665823] renesas-gbeth 15c30000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[ 4255.776289] OOM killer enabled.
[ 4255.779431] Restarting tasks ... done.
[ 4255.783936] random: crng reseeded on system resumption
[ 4255.789595] PM: suspend exit
[ 4255.793070] PM: suspend entry (s2idle)
[ 4255.797174] Filesystems sync: 0.000 seconds
[ 4255.801734] Freezing user space processes
[ 4255.806464] Freezing user space processes completed (elapsed 0.004 seconds)
[ 4255.813431] OOM killer disabled.
[ 4255.816644] Freezing remaining freezable tasks
[ 4256.055857] Freezing remaining freezable tasks completed (elapsed 0.234 seconds)
[ 4256.063427] printk: Suspending console(s) (use no_console_suspend to debug)
[ 4256.101633] gpio-keys keys: failed to configure IRQ 51 as wakeup source: -38
[ 4256.101683] gpio-keys keys: PM: dpm_run_callback(): platform_pm_suspend returns -38
[ 4256.101755] gpio-keys keys: PM: failed to suspend: error -38
[ 4256.101792] PM: Some devices failed to suspend, or early wake event detected
[ 4256.103095] renesas-gbeth 15c30000.ethernet eth0: configuring for phy/rgmii-id link mode
[ 4256.107110] dwmac4: Master AXI performs fixed burst length
[ 4256.107192] renesas-gbeth 15c30000.ethernet eth0: No Safety Features support found
[ 4256.107254] renesas-gbeth 15c30000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[ 4256.215317] OOM killer enabled.
[ 4256.218475] Restarting tasks ... done.
[ 4256.222866] random: crng reseeded on system resumption
[ 4256.228101] PM: suspend exit
[ 4259.113502] renesas-gbeth 15c30000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
root@smarc-rzg3e:~# [ 4260.514428] 8021q: 802.1Q VLAN Support v1.8
[ 4260.518689] 8021q: adding VLAN 0 to HW filter on device eth0
root@smarc-rzg3e:~#
Cheers,
Biju
