On Wed, Dec 02, 2020 at 12:01:49PM +0100, Han-Wen Nienhuys wrote:
> > I'm not sure if the long-term goal is to have this opaque unit-test
> > program or not. If it is, I was likewise going to suggest that its
> > ad-hoc output be replaced with TAP. But it looks like on your branch
> > that "test-tool reftable" does not produce output at all. So I may be a
> > bit behind on what the current state and forward plans are...
>
> The most important requirement is that something fails if the
> unittests don't work. I surmised that that meant running tests from
> test-helper in some way, so this is what happens now. Looking for
> "unit.?test" across the git codebase didn't turn up much info. Happy
> to explore other solutions if you can give me pointers.
Normally we do a combination of:
- git plumbing exposes scriptable commands, and we make sure those
work. This is a much coarser-grained unit than testing individual C
functions, but produces resilient tests because that interface is
user-visible and stable (and in fact seeing test breakages is often
a sign that one will be breaking real users).
These are obviously driven by shell script tests emulating what
users might run.
- for unit tests of individual data types where it's not appropriate
to have a user-facing command, we often add a test-tool helper that
exposes specific functions (e.g., t/helper/test-date.c exposes date
parsing and formatting routines, test-oid-array.c exercises methods
of that data type, etc).
The C parts of these are usually generic, and then are driven by
shell scripts providing the actual data in individual tests (and
handling success/failure, skipping tests, etc).
This is still coarser than you might get unit-testing inside C.
E.g., you could not generally check the difference between passing
an empty array vs NULL to a function. But our philosophy has
generally been _not_ to test at that level. The C interfaces are
internal, and Git is not that big a project. If there's a function
whose caller does something unexpected, it's usually easier to fix
the caller and add a test that triggers the caller's code path.
I agree that a test that simply runs a bunch of C code and either exits
with failure or success is better than nothing, in the sense of finding
tests. And wrapping that with a single test_expect_success does that.
But it's unfortunate that we get none of the fine-grained control that
the test suite provides, nor much support in debugging failing tests.
One middle ground would be for a battery of C tests to output
TAP-compatible output (outputting "ok 1 - foo works", and "not ok 2 -
bar does not work", etc). That at least gives more info on what fails,
and does it in a way that the rest of the test suite can interpret
(though not manipulate).
-Peff
