Patrick Steinhardt <ps@xxxxxx> writes:
> It is quite easy for the list of integration tests to go out-of-sync
> without anybody noticing. Introduce a new configure-time check that
> verifies that all tests are wired up properly.
>
> Signed-off-by: Patrick Steinhardt <ps@xxxxxx>
> ---
> t/meson.build | 36 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
>
> diff --git a/t/meson.build b/t/meson.build
> index 9e676e69363ed6311426500d98fe281e30d26bcb..f1fbc6ae179079f4d5d86f9a60956fad84d0495c 100644
> --- a/t/meson.build
> +++ b/t/meson.build
> @@ -1092,6 +1092,42 @@ integration_tests = [
> 't9903-bash-prompt.sh',
> ]
>
> +# Sanity check that we are not missing any tests present in 't/'. This check
> +# only runs once at configure time and is thus best-effort, only. It is
> +# sufficient to catch missing test suites in our CI though.
> +foreach glob, tests : {
> + 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
> + 'unit-tests/t-*.c': unit_test_programs,
> + 'unit-tests/u-*.c': clar_test_suites,
> +}
> + actual_tests = run_command(shell, '-c', 'ls ' + glob,
> + check: true,
> + env: script_environment,
> + ).stdout().strip().split('\n')
> +
> + if tests != actual_tests
> + missing_tests = [ ]
> + foreach actual_test : actual_tests
> + if actual_test not in tests
> + missing_tests += actual_test
> + endif
> + endforeach
> + if missing_tests.length() > 0
> + error('Missing tests:\n\n - ' + '\n - '.join(missing_tests))
This gives nice output:
$ touch t/unit-tests/u-bar.c t/unit-tests/u-foo.c
$ meson setup builddir --reconfigure
The Meson build system
Version: 1.4.1
[snip]
Configuring update.sample using configuration
Configuring exclude using configuration
t/meson.build:1116:6: ERROR: Problem encountered: Missing tests:
- unit-tests/u-bar.c
- unit-tests/u-foo.c
A full log can be found at git/builddir/meson-logs/meson-log.txt
But I think the error message is a little bit confusing. It sounds like
the test file is missing, but it's the configuration of the test that is
missing.
Now I've realized it hard to write a good error message here. But I
would suggest something like "Tests files found, but not configured".
> + endif
> +
> + superfluous_tests = [ ]
> + foreach integration_test : tests
> + if integration_test not in actual_tests
> + superfluous_tests += integration_test
> + endif
> + endforeach
> + if superfluous_tests.length() > 0
> + error('Superfluous tests:\n\n - ' + '\n - '.join(superfluous_tests))
Also here I would suggest different error message, maybe something like
"Tests configured, but file not found"
--
Toon
