Hi Josh
On 08/08/2023 00:07, Josh Steadmon wrote:
Reviewers can help this series progress by discussing whether it's
acceptable to rely on `prove` as a test harness for unit tests. We
support this for the current shell tests suite, but it is not strictly
required.
If possible it would be good to be able to run individual test programs
without a harness as we can for our integration tests. For running more
than one test program in parallel I think it is fine to require a harness.
I don't have a strong preference for which harness we use so long as it
provides a way to (a) run tests that previously failed tests first and
(b) run slow tests first. I do have a strong preference for using the
same harness for both the unit tests and the integration tests so
developers don't have to learn two different tools. Unless there is a
problem with prove it would probably make sense just to keep using that
as the project test harness.
TODOs remaining:
- Discuss pre-existing harnesses for the current test suite
- Figure out how to evaluate frameworks on additional OSes such as *BSD
and NonStop
We have .cirrus.yml in tree which I think gitgitgadget uses to run our
test suite on freebsd so we could leverage that for the unit tests. As
for NonStop I think we'd just have to rely on Randall running the tests
as he does now for the integration tests.
Changes in v5:
- Add comparison point "License".
- Discuss feature priorities
- Drop frameworks:
- Incompatible licenses: libtap, cmocka
- Missing source: MyTAP
- No TAP support: µnit, cmockery, cmockery2, Unity, minunit, CUnit
- Drop comparison point "Coverage reports": this can generally be
handled by tools such as `gcov` regardless of the framework used.
- Drop comparison point "Inline tests": there didn't seem to be
strong interest from reviewers for this feature.
- Drop comparison point "Scheduling / re-running": this was not
supported by any of the main contenders, and is generally better
handled by the harness rather than framework.
- Drop comparison point "Lazy test planning": this was supported by
all frameworks that provide TAP output.
These changes all sound sensible to me
The trimmed down table is most welcome. The custom implementation
supports partial parallel execution. It shouldn't be too difficult to
add some signal handling to it depending on what we want it to do.
It sounds like we're getting to the point where we have pinned down our
requirements and the available alternatives well enough to make a decision.
Best Wishes
Phillip
