On 2024.07.31 11:04, Patrick Steinhardt wrote: > Hi, > > there's been some discussion around extending our unit testing framework > to avoid duplication when declaring test functions. Right now, a testing > function has to be declared and then wired up via the test's main > function, which can be a bit annoying. In the thread, René proposes an > alternative that gets rid of this duplication by using macros. And while > that does solve the issue, there were some concerns about things being > too much "magic" while at the same time not being flexible enough. > > Part of the discussion revolved around whether we maybe want to have a > proper unit testing framework in our codebase instead of reinventing the > wheel. As I quite liked the "clar" [2] testing framework from back when > I was still developing libgit2 I proposed it as a possible alternative. > This patch series wires up the clar framework as a proof of concept and > converts the strvec test suite to use it. > > The magic to avoid the above code duplication is quite self-contained in > a "generate.py" file. This script extracts function declarations from > all unit test suites and then writes those into a "clar.suite" header > file. All that one needs to do is thus to declare a function with a > specific name "test_<suite>__<name>" and then everything else gets wired > up automatically. > > Whether this is better than the solution proposed by René is probably a > matter of taste. While I'm not a huge fan of the macro-based solution, > I don't want to veto it either (not that I'd have that power anyway). So > please, you should rather read this as a proof of concept to see how > alternatives could look like such that we have a better picture of where > we want to end up. > > Some random thoughts: > > - The mandated Python dependency is suboptimal in my opinion. > Rewriting the script in e.g. Perl should be easy enough though, it's > no rocket science. > > - I prefer that the proposed solution results in a single binary as > compared to one binary per test system. Does clar allow running test functions in parallel? With multiple binaries, we can at least run independent tests in parallel (although right now the unit tests are fewer and so much faster than the shell tests that it's hardly noticeable). > - The clar gives us the ability to pick which tests to run via command > line parameters, which I personally like more than picking the > specific binary to run. Yes this is a nice improvement. > - The clar replaces some test assertions that we already have. They > feel a bit more mature, but overall there aren't all that many > assertions available. If we wanted to pick it up, then we'd likely > have to add some more wrappers. > > - The clar uses longjmp instead of manually having to `return` from > functions in case there was an assertion failure. This is easier to > work with in my opinion. > > Also, note that I only tested this on my Linux machine. I have no clue > whether this works as-is on Windows, but I do know that libgit2 tests > run on Linux, macOS and Windows. So it should work in theory, it's just > a matter of polishing this series. > > I'm happy to hear your thoughts on this, even if it ultimately ends up > being shot down. As part of the original unit-test series, I wrote a comparison between different frameworks: Documentation/technical/unit-tests.txt, poorly rendered at [1]. Could you add a row to the table evaluating clar on the individual points there? [1] https://git-scm.com/docs/unit-tests#framework-selection > Patrick > > [1]: <85b6b8a9-ee5f-42ab-bcbc-49976b30ef33@xxxxxx> > [2]: https://github.com/clar-test/clar > > Patrick Steinhardt (3): > t: import the clar unit testing framework > Makefile: wire up the clar unit testing framework > t/unit-tests: convert strvec tests to use clar > > .gitignore | 1 + > Makefile | 36 +- > t/Makefile | 1 + > t/unit-tests/.gitignore | 3 + > t/unit-tests/clar/.github/workflows/ci.yml | 23 + > t/unit-tests/clar/COPYING | 15 + > t/unit-tests/clar/README.md | 329 ++++++++ > t/unit-tests/clar/clar.c | 842 +++++++++++++++++++++ > t/unit-tests/clar/clar.h | 173 +++++ > t/unit-tests/clar/clar/fixtures.h | 50 ++ > t/unit-tests/clar/clar/fs.h | 522 +++++++++++++ > t/unit-tests/clar/clar/print.h | 211 ++++++ > t/unit-tests/clar/clar/sandbox.h | 154 ++++ > t/unit-tests/clar/clar/summary.h | 143 ++++ > t/unit-tests/clar/generate.py | 267 +++++++ > t/unit-tests/clar/test/.gitignore | 5 + > t/unit-tests/clar/test/Makefile | 39 + > t/unit-tests/clar/test/clar_test.h | 16 + > t/unit-tests/clar/test/main.c | 40 + > t/unit-tests/clar/test/main.c.sample | 27 + > t/unit-tests/clar/test/resources/test/file | 1 + > t/unit-tests/clar/test/sample.c | 84 ++ > t/unit-tests/{t-strvec.c => strvec.c} | 124 ++- > t/unit-tests/unit-test.c | 16 + > t/unit-tests/unit-test.h | 3 + > 25 files changed, 3041 insertions(+), 84 deletions(-) > create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml > create mode 100644 t/unit-tests/clar/COPYING > create mode 100644 t/unit-tests/clar/README.md > create mode 100644 t/unit-tests/clar/clar.c > create mode 100644 t/unit-tests/clar/clar.h > create mode 100644 t/unit-tests/clar/clar/fixtures.h > create mode 100644 t/unit-tests/clar/clar/fs.h > create mode 100644 t/unit-tests/clar/clar/print.h > create mode 100644 t/unit-tests/clar/clar/sandbox.h > create mode 100644 t/unit-tests/clar/clar/summary.h > create mode 100755 t/unit-tests/clar/generate.py > create mode 100644 t/unit-tests/clar/test/.gitignore > create mode 100644 t/unit-tests/clar/test/Makefile > create mode 100644 t/unit-tests/clar/test/clar_test.h > create mode 100644 t/unit-tests/clar/test/main.c > create mode 100644 t/unit-tests/clar/test/main.c.sample > create mode 100644 t/unit-tests/clar/test/resources/test/file > create mode 100644 t/unit-tests/clar/test/sample.c > rename t/unit-tests/{t-strvec.c => strvec.c} (54%) > create mode 100644 t/unit-tests/unit-test.c > create mode 100644 t/unit-tests/unit-test.h > > -- > 2.46.0.dirty >
