On Tue, 5 Apr 2022 15:48:05 +0200 Vincent Whitchurch <vincent.whitchurch@xxxxxxxx> wrote: > On Sun, Mar 20, 2022 at 06:02:53PM +0100, Jonathan Cameron wrote: > > Very interesting bit of work. My current approach for similar testing > > is to write a qemu model for the hardware, but that currently > > requires carefully crafted tests. Most of the time I'm only doing > > that to verify refactoring of existing drivers. > > Thank you for taking a look! > > > One thing that makes me nervous here is the python element though > > as I've not written significant python in about 20 years. > > That is going to be a burden for kernel developers and maintainers... > > Nothing quite like badly written tests to make for a mess in the long run > > and I suspect my python for example would be very very badly written :) > > There's a bunch of static checkers to ensure that the code follows some > basic guidelines, and CI can check that the tests work consistently, and > also calculate metrics such as test execution time and code coverage, so > even non-idiomatic Python in the tests wouldn't be entirely broken. > > And unlike driver code, if the tests for a particular driver later do > turn out to be bad (in what way?), we could just throw those particular > tests out without breaking anybody's system. True. Though CI test triage folk may disagree ;) > > > Cut and paste will of course get us a long way... > > Isn't some amount of copy/paste followed by modification to be expected > even if the framework is written in say C (just as there's already > copy/paste + modification involved when writing drivers)? > > As for the core logic of individual driver tests excluding the framework > bits, I have a hard time imagining what Python syntax looks like to > someone with no knowledge of Python, so yes, I guess it's going to be > harder to review. I messed around the other day with writing tests for drivers/staging/iio/cdc/ad7746.c and wasn't "too bad" and was useful for verifying some refactoring (and identified a possible precision problem in some integer approximation of floating point calcs) I'll try and find time to flesh that test set out more in the near future and post it so you can see how bad my python is. It amused my wife if nothing else :) However a future project is to see if I can use this to hook up the SPDM attestation stack via mctp over i2c - just because I like to live dangerously :) For IIO use more generally we need a sensible path to SPI (and also platform drivers). For my day job I'd like to mess around with doing PCI devices as well. The PCI DOE support for example would be nice to run against a test set that doesn't involve spinning up QEMU. DOE driver support: https://lore.kernel.org/all/20220330235920.2800929-1-ira.weiny@xxxxxxxxx/ Effort wise, it's similar effort to hacking equivalent in QEMU but with the obvious advantage of being in tree and simpler for CI systems etc to use. It would be nice to only have to use QEMU for complex system CI tests like the ones we are doing for CXL. > > > I dream of a world where every driver is testable by people with out hardware > > but I fear it may be a while yet. Hopefully this will get us a little > > closer! > > > > I more or less follow what is going on here (good docs btw in the earlier > > patch definitely helped). > > > > So far I'm thoroughly in favour of road test subject to actually being > > able to review the tests or getting sufficient support to do so. > > It's a 'how to scale it' question really... > > Would rewriting the framework in C and forcing tests to be written in > that language mean that maintainers would be able to review tests > without external support? I was wondering that. If we stayed in python I think we'd definitely want someone to be the 'roadtester/tests' maintainer (or group of maintainers) and their Ack to be expected for all tests we upstream. Idea being they'd sanity check correct use of framework and just how bad the python code us C developers are writing is ;) However, we'd still need a good chunk of that 'framework' use review even if doing this in C. Anyhow, very promising bit of work. Thanks, Jonathan
