Re: [Automated-testing] [RFC] Test catalog template

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Tim,

On Tue, Oct 15, 2024 at 12:01 PM Bird, Tim <Tim.Bird@xxxxxxxx> wrote:
>
> > -----Original Message-----
> > From: automated-testing@xxxxxxxxxxxxxxxxxxxxxx <automated-testing@xxxxxxxxxxxxxxxxxxxxxx> On Behalf Of Don Zickus
> > Hi,
> >
> > At Linux Plumbers, a few dozen of us gathered together to discuss how
> > to expose what tests subsystem maintainers would like to run for every
> > patch submitted or when CI runs tests.  We agreed on a mock up of a
> > yaml template to start gathering info.  The yaml file could be
> > temporarily stored on kernelci.org until a more permanent home could
> > be found.  Attached is a template to start the conversation.
> >
> Don,
>
> I'm interested in this initiative.  Is discussion going to be on a kernel mailing
> list, or on this e-mail, or somewhere else?

I was going to keep it on this mailing list.  Open to adding other
lists or moving it.

>
> See a few comments below.
>
> > Longer story.
> >
> > The current problem is CI systems are not unanimous about what tests
> > they run on submitted patches or git branches.  This makes it
> > difficult to figure out why a test failed or how to reproduce.
> > Further, it isn't always clear what tests a normal contributor should
> > run before posting patches.
> >
> > It has been long communicated that the tests LTP, xfstest and/or
> > kselftests should be the tests  to run.
> Just saying "LTP" is not granular enough.  LTP has hundreds of individual
> test programs, and it would be useful to specify the individual tests
> from LTP that should be run per sub-system.

Agreed.  Just reiterating what Greg has told me.

>
> I was particularly intrigued by the presentation at Plumbers about
> test coverage.  It would be nice to have data (or easily replicable
> methods) for determining the code coverage of a test or set of
> tests, to indicate what parts of the kernel are being missed
> and help drive new test development.

It would be nice.  I see that as orthogonal to this effort for now.
But I think this might be a good step towards that idea.

>
> > However, not all maintainers
> > use those tests for their subsystems.  I am hoping to either capture
> > those tests or find ways to convince them to add their tests to the
> > preferred locations.
> >
> > The goal is for a given subsystem (defined in MAINTAINERS), define a
> > set of tests that should be run for any contributions to that
> > subsystem.  The hope is the collective CI results can be triaged
> > collectively (because they are related) and even have the numerous
> > flakes waived collectively  (same reason) improving the ability to
> > find and debug new test failures.  Because the tests and process are
> > known, having a human help debug any failures becomes easier.
> >
> > The plan is to put together a minimal yaml template that gets us going
> > (even if it is not optimized yet) and aim for about a dozen or so
> > subsystems.  At that point we should have enough feedback to promote
> > this more seriously and talk optimizations.
>
> Sounds like a good place to start.  Do we have some candidate sub-systems
> in mind?  Has anyone volunteered to lead the way?

At our meeting, someone suggested Kunit as it was easy to understand
for starters and then add a few other volunteer systems in.  I know we
have a few maintainers who can probably help us get started.  I think
arm and media were ones thrown about at our meeting.

>
> >
> > Feedback encouraged.
> >
> > Cheers,
> > Don
> >
> > ---
> > # List of tests by subsystem
> > #
> > # Tests should adhere to KTAP definitions for results
> > #
> > # Description of section entries
> > #
> > #  maintainer:    test maintainer - name <email>
> > #  list:                mailing list for discussion
> > #  version:         stable version of the test
> > #  dependency: necessary distro package for testing
> > #  test:
> > #    path:            internal git path or url to fetch from
> > #    cmd:            command to run; ability to run locally
> > #    param:         additional param necessary to run test
> > #  hardware:      hardware necessary for validation
>
> Is this something new in MAINTAINERS, or is it a separate file?

For now a separate file.  It isn't clear where this could go long
term.  The thought was to gather data to see what is necessary first.
Long term it will probably stay a separate file. *shrugs*

>
> > #
> > # Subsystems (alphabetical)
> >
> > KUNIT TEST:
> >   maintainer:
> >     - name: name1
> >       email: email1
> >     - name: name2
> >       email: email2
> >   list:
> >   version:
> >   dependency:
> >     - dep1
> >     - dep2
> >   test:
> >     - path: tools/testing/kunit
> >       cmd:
> >       param:
> >     - path:
> >       cmd:
> >       param:
> >   hardware: none
>
> Looks OK so far - it'd be nice to have a few concrete examples.

Fair enough.  Let me try and work on some.

Cheers,
Don

>  -- Tim
>






[Index of Archives]     [Linux Samsung SoC]     [Linux Rockchip SoC]     [Linux Actions SoC]     [Linux for Synopsys ARC Processors]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]


  Powered by Linux