Hi David, On Wed, Nov 20, 2024 at 3:16 AM David Gow <davidgow@xxxxxxxxxx> wrote: > > On Thu, 7 Nov 2024 at 01:01, Donald Zickus <dzickus@xxxxxxxxxx> wrote: > > > > Hi, > > > > Thanks for the feedback. I created a more realistic test.yaml file to > > start (we can split it when more tests are added) and a parser. I was > > going to add patch support as input to mimic get_maintainers.pl > > output, but that might take some time. For now, you have to manually > > select a subsystem. I will try to find space on kernelci.org to grow > > this work but you can find a git tree here[0]. > > > > From the README.md > > """ > > An attempt to map kernel subsystems to kernel tests that should be run > > on patches or code by humans and CI systems. > > > > Examples: > > > > Find test info for a subsystem > > > > ./get_tests.py -s 'KUNIT TEST' --info > > > > Subsystem: KUNIT TEST > > Maintainer: > > David Gow <davidgow@xxxxxxxxxx> > > Mailing List: None > > Version: None > > Dependency: ['python3-mypy'] > > Test: > > smoke: > > Url: None > > Working Directory: None > > Cmd: ./tools/testing/kunit/kunit.py > > Env: None > > Param: run --kunitconfig lib/kunit > > Hardware: arm64, x86_64 > > > > Find copy-n-pastable tests for a subsystem > > > > ./get_tests.py -s 'KUNIT TEST' > > > > ./tools/testing/kunit/kunit.pyrun --kunitconfig lib/kunit > > """ > > > > Is this aligning with what people were expecting? > > > > > Awesome! I've been playing around a bit with this, and I think it's an > excellent start. > > There are definitely some more features I'd want in an ideal world > (e.g., configuration matrices, etc), but this works well enough. Yeah, I was trying to nail down the usability angle first before expanding with bells and whistles. I would like to think the yaml file is flexible enough to handle those features though?? > > I've been playing around with a branch which adds the ability to > actually run these tests, based on the 'run_checks.py' script we use > for KUnit: > https://github.com/sulix/test-catalog/tree/runtest-wip Thanks! > > In particular, this adds a '-r' option which runs the tests for the > subsystem in parallel. This largely matches what I was doing manually > — for instance, the KUnit section in test.yaml now has three different > tests, and running it gives me this result: > ../test-catalog/get_tests.py -r -s 'KUNIT TEST' > Waiting on 3 checks (kunit-tool-test, uml, x86_64)... > kunit-tool-test: PASSED > x86_64: PASSED > uml: PASSED Interesting. Originally I was thinking this would be done serially. I didn't think tests were safe enough to run in parallel. I am definitely open to this. My python isn't the best, but I think your PR looks reasonable. > > (Obviously, in the real world, I'd have more checks, including other > architectures, checkpatch, etc, but this works as a proof-of-concept > for me.) > > I think the most interesting questions will be: > - How do we make this work with more complicated dependencies > (containers, special hardware, etc)? I was imagining a 'hw-requires' type line to handle the hardware requests as that seemed natural for a lot of the driver work. Run a quick check before running the test to see if the required hw is present or not and bail if it isn't. The containers piece is a little trickier and ties into the test environment I think. The script would have to create an environment and inject the tests into the environment and run them. I would imagine some of this would have to be static as the setup is complicated. For example, a 'container' label would execute custom code to setup a test environment inside a container. Open to ideas here. > - How do we integrate it with CI systems — can we pull the subsystem > name for a patch from MAINTAINERS and look it up here? There are two thoughts. First is yes. As a developer you probably want to run something like 'get_maintainers.sh <patch> | get_tests.py -s -' or something to figure out what variety of tests you should run before posting. And a CI system could probably do something similar. There is also another thought, you already know the subsystem you want to test. For example, a patch is usually written for a particular subsystem that happens to touch code from other subsystems. You primarily want to run it against a specified subsystem. I know Red Hat's CKI will run against a known subsystem git-tree and would fall into this category. While it does leave a gap in other subsystem testing, sometimes as a human you already know running those extra tests is mostly a no-op because it doesn't really change anything. > - What about things like checkpatch, or general defconfig build tests > which aren't subsystem-specific? My initial thought is that this is another category of testing. A lot of CI tests are workload testing and have predefined configs. Whereas a generic testing CI system (think 0-day) would focus on those types of testing. So I would lean away from those checks in this approach or we could add a category 'general' too. I do know checkpatch rules vary from maintainer to maintainer. > - How can we support more complicated configurations or groups of > configurations? Examples? > - Do we add support for specific tools and/or parsing/combining output? Examples? I wasn't thinking of parsing test output, just providing what to run as a good first step. My initial thought was to help nudge tests towards the KTAP output?? > > But I'm content to keep playing around with this a bit more for now. Thank you! Please do! Cheers, Don
