On Mon, Jun 14, 2021 at 11:33:12AM +0200, Ævar Arnfjörð Bjarmason wrote:
> > I think breaking the test suite is objectively worse than having a few
> > extra files in the output directory, but to each his own.
>
> We've got both in-tree and out-tree things that rely on e.g. the
> *.counts in that directory to have a 1=1 mapping with "real"
> tests. E.g. "make aggregate-results".
Indeed. With Felipe's original patch, the "test" target (but not
"prove") in t/Makefile will report, whether you set
TEST_OUTPUT_DIRECTORY or not:
failed test(s): t1234 t2345
fixed 0
success 23243
failed 2
broken 221
total 23647
though curiously it doesn't exit non-zero back to make (usually we'd
also see the failures from the individual make targets, and barf there).
> diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
> index 2c6e34b9478..29bf67d49bf 100755
> --- a/t/t0000-basic.sh
> +++ b/t/t0000-basic.sh
> @@ -76,6 +76,12 @@ _run_sub_test_lib_test_common () {
> # this variable, so we need a stable setting under which to run
> # the sub-test.
> sane_unset HARNESS_ACTIVE &&
> +
> + # These tests should emit no metrics or output that
> + # would normally go in the "test-results" directory.
> + TEST_NO_RESULTS_OUTPUT=1 &&
> + export TEST_NO_RESULTS_OUTPUT &&
I'm OK with this general approach. I do think it would be nice if we let
the environment supersede the on-disk GIT-BUILD-OPTIONS, which IMHO is
the real root of the problem (and possibly others), but that may be more
challenging to get right (I posted a patch earlier, but it does rely on
stuffing all of "set" into a variable, which makes me concerned some
less-able shells may complain).
It also means that t0000 can't test the results output (since we don't
write it), but I assume we don't do that now (I didn't actually try
running with your patch).
> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index 54938c64279..9e9696a3185 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -252,8 +252,14 @@ TEST_STRESS_JOB_SFX="${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}"
> TEST_NAME="$(basename "$0" .sh)"
> TEST_NUMBER="${TEST_NAME%%-*}"
> TEST_NUMBER="${TEST_NUMBER#t}"
> -TEST_RESULTS_DIR="$TEST_OUTPUT_DIRECTORY/test-results"
> -TEST_RESULTS_BASE="$TEST_RESULTS_DIR/$TEST_NAME$TEST_STRESS_JOB_SFX"
> +if test -n "$TEST_NO_RESULTS_OUTPUT"
> +then
> + TEST_RESULTS_DIR=/dev/null
> + TEST_RESULTS_BASE=/dev/null
> +else
> + TEST_RESULTS_DIR="$TEST_OUTPUT_DIRECTORY/test-results"
> + TEST_RESULTS_BASE="$TEST_RESULTS_DIR/$TEST_NAME$TEST_STRESS_JOB_SFX"
> +fi
I wondered about this use of /dev/null, since we'd generally use this as
a directory, and writing to "/dev/null/foo" is going to throw an error.
But...
> TRASH_DIRECTORY="trash directory.$TEST_NAME$TEST_STRESS_JOB_SFX"
> test -n "$root" && TRASH_DIRECTORY="$root/$TRASH_DIRECTORY"
> case "$TRASH_DIRECTORY" in
> @@ -1124,7 +1130,7 @@ test_done () {
>
> finalize_junit_xml
>
> - if test -z "$HARNESS_ACTIVE"
> + if test -z "$HARNESS_ACTIVE$TEST_NO_RESULTS_OUTPUT"
> then
> mkdir -p "$TEST_RESULTS_DIR"
...here we would never look at those variables at all, so it is just a
sentinel that would let us know the assumption has been violated.
We do look at them elsewhere, though (in --tee as you noted, and I think
for --stress). I'd prefer to notice the "no results" flag explicitly
there and report something sensible, rather than getting:
mkdir: cannot create directory ‘/dev/null’: Not a directory
or similar.
-Peff
