On 2023.08.14 14:29, Phillip Wood wrote: > Hi Josh > > On 08/08/2023 00:07, Josh Steadmon wrote: > > > > Reviewers can help this series progress by discussing whether it's > > acceptable to rely on `prove` as a test harness for unit tests. We > > support this for the current shell tests suite, but it is not strictly > > required. > > If possible it would be good to be able to run individual test programs > without a harness as we can for our integration tests. For running more than > one test program in parallel I think it is fine to require a harness. Sounds good. This is working in v6 which I hope to send to the list soon. > I don't have a strong preference for which harness we use so long as it > provides a way to (a) run tests that previously failed tests first and (b) > run slow tests first. I do have a strong preference for using the same > harness for both the unit tests and the integration tests so developers > don't have to learn two different tools. Unless there is a problem with > prove it would probably make sense just to keep using that as the project > test harness. To be clear, it sounds like both of these can be done with `prove` (using the various --state settings) without any further support from our unit tests, right? I see that we do have a "failed" target for re-running integration tests, but that relies on some test-lib.sh features that currently have no equivalent in the unit test framework. > > TODOs remaining: > > - Discuss pre-existing harnesses for the current test suite > > - Figure out how to evaluate frameworks on additional OSes such as *BSD > > and NonStop > > We have .cirrus.yml in tree which I think gitgitgadget uses to run our test > suite on freebsd so we could leverage that for the unit tests. As for > NonStop I think we'd just have to rely on Randall running the tests as he > does now for the integration tests. Thanks for the pointer. I've updated .cirrus.yml (as well as the GitHub CI configs) for v6. > > Changes in v5: > > - Add comparison point "License". > > - Discuss feature priorities > > - Drop frameworks: > > - Incompatible licenses: libtap, cmocka > > - Missing source: MyTAP > > - No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit > > - Drop comparison point "Coverage reports": this can generally be > > handled by tools such as `gcov` regardless of the framework used. > > - Drop comparison point "Inline tests": there didn't seem to be > > strong interest from reviewers for this feature. > > - Drop comparison point "Scheduling / re-running": this was not > > supported by any of the main contenders, and is generally better > > handled by the harness rather than framework. > > - Drop comparison point "Lazy test planning": this was supported by > > all frameworks that provide TAP output. > > These changes all sound sensible to me > > The trimmed down table is most welcome. The custom implementation supports > partial parallel execution. It shouldn't be too difficult to add some signal > handling to it depending on what we want it to do. > > It sounds like we're getting to the point where we have pinned down our > requirements and the available alternatives well enough to make a decision. Yes, v6 will include your TAP implementation (I assume you are still OK if I include your patch in this series?). > Best Wishes > > Phillip >
