Re: [libgpiod][PATCH] tools: tests: replace egrep with grep -E

[Kent Gibson <warthog618@xxxxxxxxx>]

On Fri, Jun 02, 2023 at 05:33:08PM +0200, Bartosz Golaszewski wrote:
> On Fri, Jun 2, 2023 at 3:26 PM Kent Gibson <warthog618@xxxxxxxxx> wrote:
> >
> > On Fri, Jun 02, 2023 at 03:10:18PM +0200, Bartosz Golaszewski wrote:
> > > On Fri, Jun 2, 2023 at 12:34 PM Kent Gibson <warthog618@xxxxxxxxx> wrote:
> > > >
> > > > On Fri, Jun 02, 2023 at 10:13:35AM +0800, Kent Gibson wrote:
> > > > >
> > > > > On a related(??) note, I'm occasionally seeing Oopses when testing this
> > > > > - when creating a basic sim with a shell script, not when deleting it.
> > > > > In one case after a fresh reboot and on creating the first sim, so it
> > > > > looks to be purely the construction.  Yay :-(.
> > > > >
> > > >
> > > > I had thought it would be difficult to reproduce this and so difficult
> > > > to bisect.  Fortunately(??) not.  If I run my setup and cleanup scripts[1]
> > > > in a tight loop it occurs very readily.  Haven't bisected it yet, but did
> > > > test it on 6.1-rc1 and it Oopsed there too, so I would need to go back
> > > > further.  What was the initial release containing gpio-sim?
> > > >
> > > > The sim setp is pretty simple - a couple of banks each with a few lines
> > > > and hogs.
> > > >
> > > > Could you confirm that you can repeat the problem?
> > > > Otherwise I might start thinking there is something broken in my test
> > > > environment.
> > > >
> > > > Btw, the loop script is:
> > > >
> > > > #!/bin/env bash
> > > > for (( ; ; ))
> > > > do
> > > >         echo "create sim..."
> > > >         ./basic_sim.sh
> > > >         echo "destroy sim..."
> > > >         ./clean_sims.sh
> > > > done
> > > >
> > > > Cheers,
> > > > Kent.
> > > > [1] https://github.com/warthog618/gpiosim-rs
> > > >
> > >
> > > With this script I've been able to trigger an issue but it looks
> > > different from yours: https://pastebin.com/cbsgT2ae
> > >
> >
> > Looks similar to me.
> > I would assume that is the same issue - even if the  particulars of the
> > crash differ.  If you can fix that and my problem remains then we can be
> > sure they are distinct.
> >
> > I've been doing a coarse bisect to see how far back this goes -
> > basically looking for a known good.
> > 5.18 crashes, but it crashed hard, so no syslog.  It did run considerably
> > longer before crashing, so that could be different issue masked by the
> > other (later?) one.
> >
> > Moving on to subsequent releases...
> >
> > Cheers,
> > Kent.
> 
> I managed to trigger a different crash: https://pastebin.com/6Gx29vHB
> 
> This one happened in gpio-sim:
> 
> $ ./scripts/faddr2line drivers/gpio/gpio-sim.o
> gpio_sim_make_bank_swnode+0x12f/0x220
> gpio_sim_make_bank_swnode+0x12f/0x220:
> gpio_sim_make_line_names at
> /home/brgl/workspace/yocto-gpio-sim-crash/build/linux/drivers/gpio/gpio-sim.c:725
> (inlined by) gpio_sim_make_bank_swnode at
> /home/brgl/workspace/yocto-gpio-sim-crash/build/linux/drivers/gpio/gpio-sim.c:837
> 
> But to me it looks like some memory corruption if the stack traces are
> so random... Where is Rust when you need it? :)
> 

Makes sense.  I'm seeing different behaviour from differnet releases,
so there may be multiple issues here, and some may've been fixed or
introduced at different points.
Or, as you suggest, it could be a corruption that just exhibits
differently depending on what ends up getting corrupted.
Either way, bisecting this is problematic.

e.g.
5.18 crashed hard after a while.
5.19 ran for quite a while - but then crashed when I killed the script.
6.0 behaved similar to 5.19.
6.1 looks to be where the current issue was introduced - that fails
almost immediately.
6.2 crashed hard.
6.3 failed fast like 6.1.

TBH, haven't been paying much attention to the crash details - for the
most part the backtraces are a pretty generic syscall fell in a heap as
a pointer somewhere was null.

Not sure I'm being much help with this one.
Anything specific you want me to try?
Otherwise I'll leave you to it until you do.

One other thing - is it necessary for gpio-sim to log hogs?

Jun  2 23:45:13 firefly kernel: [   39.539144] gpio-522 (hog2): hogged as input
Jun  2 23:45:13 firefly kernel: [   39.539152] gpio-528 (hog3): hogged as output/low
Jun  2 23:45:13 firefly kernel: [   39.600144] gpio-513 (hog1): hogged as output/high
Jun  2 23:45:13 firefly kernel: [   39.600365] gpio-522 (hog2): hogged as input
Jun  2 23:45:13 firefly kernel: [   39.600372] gpio-528 (hog3): hogged as output/low
Jun  2 23:45:13 firefly kernel: [   39.649662] gpio-513 (hog1): hogged as output/high
Jun  2 23:45:13 firefly kernel: [   39.650558] gpio-522 (hog2): hogged as input

For the long running cases this results in a lot of logging, which could
be causing issues in itself, and further muddying the waters.

Cheers,
Kent.