On 21/05/2021 12:14, Tony Lindgren wrote:
* Tomi Valkeinen <tomi.valkeinen@xxxxxxxxxxxxxxxx> [210521 08:45]:
On 21/05/2021 10:39, Tony Lindgren wrote:
* Tomi Valkeinen <tomi.valkeinen@xxxxxxxxxxxxxxxx> [210521 07:05]:
On 21/05/2021 08:36, Tony Lindgren wrote:
* Tomi Valkeinen <tomi.valkeinen@xxxxxxxxxxxxxxxx> [210520 08:27]:
Hi,
I've noticed that the v5.13 rcs crash randomly (but quite often) on dra76 evm
(I haven't tested other boards). Anyone else seen this problem?
I have not seen this so far and beagle-x15 is behaving for me.
Does it always happen on boot?
No, but quite often. I can't really say how often, as it's annoyingly random.
I tried to bisect, but that proved to be difficult as sometimes I get multiple (5+)
successful boots before the crash.
I tested with x15, same issue (below). So... Something in my kernel config? Or compiler?
Looks like the crash happens always very soon after (or during) probing palmas.
After about 10 reboots with your .config I'm seeing it now too on
beagle-x15. So far no luck reproducing it with omap2plus_defconfig.
I think I have an easy way to see if a kernel is good or bad, by printing
stack_not_used(current) in the first call to omap_i2c_xfer_irq(). There's a
huge drop between v5.12 and v5.13-rc1.
And interestingly, sometimes a simple printk seems to use hundreds of bytes
of stack (i.e. compare stack usage before and after the print). But not
always. So maybe the issue is somehow related to printk.
I'm bisecting.
OK sounds good to me.
Well, I found the bad commit but unfortunately it doesn't exactly point
where the issue is.
f483a3e123410bd1c78af295bf65feffb6769a98 is the first bad commit
commit f483a3e123410bd1c78af295bf65feffb6769a98
Author: Tony Lindgren <tony@xxxxxxxxxxx>
Date: Wed Mar 10 14:03:48 2021 +0200
ARM: dts: Configure simple-pm-bus for dra7 l4_per1
We can now probe interconnects with device tree only configuration
using
simple-pm-bus and genpd.
Tested-by: Kishon Vijay Abraham I <kishon@xxxxxx>
Signed-off-by: Tony Lindgren <tony@xxxxxxxxxxx>
arch/arm/boot/dts/dra7-l4.dtsi | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
The difference is clear, though. With
9a75368b6426739e8b798592f084cb682d760568, which is the last good commit,
when I print the stack usage with stack_not_used() in three different
places in omap_i2c_xfer_irq(), I get always prints roughly like:
STACK FREE omap_i2c_xfer_irq: 2972, 2972, 2972
And these repeat exactly the same for each call to omap_i2c_xfer_irq (at
least during palmas probe).
With the bad commit the situation is different. The first call to
omap_i2c_xfer_irq prints:
STACK FREE omap_i2c_xfer_irq: 2024, 2024, 2024
so we're already using 1k more. But then, instead of the stack usage
staying the same, consecutive calls show increased stack usage. It
doesn't increase for each xfer call, but after about 10 calls, I'm
getting ~1800, ten calls more I see ~800, and going down to ~500.
However, with this bad commit, I don't see the empty stack going below
~500, so I don't get crashes. But going to a more recent commit, like
01d7136894410a71932096e0fb9f1d301b6ccf07, the situation is much worse.
The first print shows:
STACK FREE omap_i2c_xfer_irq: 1164, 1164, 1164
and it quickly goes to stack overflow.
Tomi
