In our current testing environment, we spend a significant amount of effort crafting end-to-end tests for error conditions that could easily be captured by unit tests (or we simply forgo some hard-to-setup and rare error conditions).Describe what we hope to accomplish by implementing unit tests, and explain some open questions and milestones. Discuss desired features for test frameworks/harnesses, and provide a preliminary comparison of several different frameworks. Signed-off-by: Josh Steadmon <steadmon@xxxxxxxxxx> Coauthored-by: Calvin Wan <calvinwan@xxxxxxxxxx> --- Documentation/Makefile | 1 + Documentation/technical/unit-tests.txt | 141 +++++++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100644 Documentation/technical/unit-tests.txt diff --git a/Documentation/Makefile b/Documentation/Makefile index b629176d7d..3f2383a12c 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -122,6 +122,7 @@ TECH_DOCS += technical/scalar TECH_DOCS += technical/send-pack-pipeline TECH_DOCS += technical/shallow TECH_DOCS += technical/trivial-merge +TECH_DOCS += technical/unit-tests SP_ARTICLES += $(TECH_DOCS) SP_ARTICLES += technical/api-index diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt new file mode 100644 index 0000000000..dac8062a43 --- /dev/null +++ b/Documentation/technical/unit-tests.txt @@ -0,0 +1,141 @@ += Unit Testing + +In our current testing environment, we spend a significant amount of effort +crafting end-to-end tests for error conditions that could easily be captured by +unit tests (or we simply forgo some hard-to-setup and rare error conditions). +Unit tests additionally provide stability to the codebase and can simplify +debugging through isolation. Writing unit tests in pure C, rather than with our +current shell/test-tool helper setup, simplifies test setup, simplifies passing +data around (no shell-isms required), and reduces testing runtime by not +spawning a separate process for every test invocation. + +We believe that a large body of unit tests, living alongside the existing test +suite, will improve code quality for the Git project. + +== Definitions + +For the purposes of this document, we'll use *test framework* to refer to +projects that support writing test cases and running tests within the context +of a single executable. *Test harness* will refer to projects that manage +running multiple executables (each of which may contain multiple test cases) and +aggregating their results. + +In reality, these terms are not strictly defined, and many of the projects +discussed below contain features from both categories. + + +== Choosing a framework & harness + +=== Desired features + +==== TAP support + +The https://testanything.org/[Test Anything Protocol] is a text-based interface +that allows tests to communicate with a test harness. It is already used by +Git's integration test suite. Supporting TAP output is a mandatory feature for +any prospective test framework. + +==== Diagnostic output + +When a test case fails, the framework must generate enough diagnostic output to +help developers find the appropriate test case in source code in order to debug +the failure. + +==== Parallel execution + +Ideally, we will build up a significant collection of unit tests cases, most +likely split across multiple executables. It will be necessary to run these +tests in parallel to enable fast develop-test-debug cycles. + +==== Vendorable or ubiquitous + +If possible, we want to avoid forcing Git developers to install new tools just +to run unit tests. So any prospective frameworks and harnesses must either be +vendorable (meaning, we can copy their source directly into Git's repository), +or so ubiquitous that it is reasonable to expect that most developers will have +the tools installed already. + +==== Maintainable / extensible + +It is unlikely that any pre-existing project perfectly fits our needs, so any +project we select will need to be actively maintained and open to accepting +changes. Alternatively, assuming we are vendoring the source into our repo, it +must be simple enough that Git developers can feel comfortable making changes as +needed to our version. + +==== Major platform support + +At a bare minimum, unit-testing must work on Linux, MacOS, and Windows. + +==== Lazy test planning + +TAP supports the notion of _test plans_, which communicate which test cases are +expected to run, or which tests actually ran. This allows test harnesses to +detect if the TAP output has been truncated, or if some tests were skipped due +to errors or bugs. + +The test framework should handle creating plans at runtime, rather than +requiring test developers to manually create plans, which leads to both human- +and merge-errors. + +==== Skippable tests + +Test authors may wish to skip certain test cases based on runtime circumstances, +so the framework should support this. + +==== Test scheduling / re-running + +The test harness scheduling should be configurable so that e.g. developers can +choose to run slow tests first, or to run only tests that failed in a previous +run. + +==== Mock support + +Unit test authors may wish to test code that interacts with objects that may be +inconvenient to handle in a test (e.g. interacting with a network service). +Mocking allows test authors to provide a fake implementation of these objects +for more convenient tests. + +==== Signal & exception handling + +The test framework must fail gracefully when test cases are themselves buggy or +when they are interrupted by signals during runtime. + +==== Coverage reports + +It may be convenient to generate coverage reports when running unit tests +(although it may be possible to accomplish this regardless of test framework / +harness support). + + +=== Comparison + +[format="csv",options="header",width="75%"] +|===== +Framework,"TAP support","Diagnostic output","Parallel execution","Vendorable / ubiquitous","Maintainable / extensible","Major platform support","Lazy test planning","Runtime- skippable tests","Scheduling / re-running",Mocks,"Signal & exception handling","Coverage reports" +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@xxxxxxxxx/[Custom Git impl.],[lime-background]#True#,[lime-background]#True#,?,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,?,?,[red-background]#False#,?,? +https://cmocka.org/[cmocka],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[lime-background]#True#,?,? +https://libcheck.github.io/check/[Check],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,? +https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#True#,[red-background]#False#,?,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,? +https://github.com/silentbicycle/greatest[Greatest],[yellow-background]#Partial#,?,?,[lime-background]#True#,[yellow-background]#Partial#,?,[yellow-background]#Partial#,?,?,[red-background]#False#,?,? +https://github.com/Snaipe/Criterion[Criterion],[lime-background]#True#,?,?,[red-background]#False#,?,[lime-background]#True#,?,?,?,[red-background]#False#,?,? +https://github.com/zorgnax/libtap[libtap],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,? +https://nemequ.github.io/munit/[µnit],?,?,?,?,?,?,?,?,?,?,?,? +https://github.com/google/cmockery[cmockery],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,? +https://github.com/lpabon/cmockery2[cmockery2],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,? +https://github.com/ThrowTheSwitch/Unity[Unity],?,?,?,?,?,?,?,?,?,?,?,? +https://github.com/siu/minunit[minunit],?,?,?,?,?,?,?,?,?,?,?,? +https://cunit.sourceforge.net/[CUnit],?,?,?,?,?,?,?,?,?,?,?,? +https://www.kindahl.net/mytap/doc/index.html[MyTAP],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,? +|===== + +== Milestones + +* Settle on final framework +* Add useful tests of library-like code +* Integrate with Makefile +* Integrate with CI +* Integrate with + https://lore.kernel.org/git/20230502211454.1673000-1-calvinwan@xxxxxxxxxx/[stdlib + work] +* Run alongside regular `make test` target -- 2.41.0.162.gfafddb0af9-goog
