To check that a git command fails and to inspect its error message we
usually execute a command like this throughout our test suite:
test_must_fail git command --option 2>output.err
Note that this command doesn't limit the redirection to the git
command, but it redirects the standard error of the 'test_must_fail'
helper function as well. This is wrong for several reasons:
- If that git command were to succeed or die for an unexpected
reason e.g. signal, then 'test_must_fail's helpful error message
would end up in the given file instead of on the terminal or in
't/test-results/$T.out', when the test is run with '-v' or
'--verbose-log', respectively.
- If the test is run with '-x', the trace of executed commands in
'test_must_fail' will go to stderr as well (except for more recent
Bash versions supporting $BASH_XTRACEFD), and then in turn will
end up in the given file.
- If the git command's error message is checked verbatim with
'test_cmp', then the '-x' trace will cause the test to fail.
- Though rather unlikely, if the git command's error message is
checked with 'test_i18ngrep', then its pattern might match
'test_must_fail's error message (or '-x' trace) instead of the
given git command's, potentially hiding a bug.
Teach the 'test_must_fail' helper the 'stderr=<file>' option to save
the executed git command's standard error to that file, so we can
avoid all the bad side effects of redirecting the whole thing's
standard error.
The same issues apply to the 'test_might_fail' helper function as
well, but since it's implemented as a thin wrapper around
'test_must_fail', no modifications were necessary and 'test_might_fail
stderr=output.err git command' will just work.
Signed-off-by: SZEDER Gábor <szeder.dev@xxxxxxxxx>
---
Notes:
I considered adding an analogous 'stdout=<file>' option as well, but in
the end haven't, because:
- 'test_must_fail' doesn't print anything to its standard output, so a
plain '>out' redirection at the end of the line doesn't have any
undesirable side effects.
- We would need four conditions to cover all possible combinations of
'stdout=' and 'stderr='. Considering the above point, I don't think
it's worth it.
t/README | 6 ++++++
t/test-lib-functions.sh | 35 +++++++++++++++++++++++++----------
2 files changed, 31 insertions(+), 10 deletions(-)
diff --git a/t/README b/t/README
index 1a1361a806..2528359e9d 100644
--- a/t/README
+++ b/t/README
@@ -430,6 +430,10 @@ Don't:
use 'test_must_fail git cmd'. This will signal a failure if git
dies in an unexpected way (e.g. segfault).
+ Don't redirect 'test_must_fail's standard error like
+ 'test_must_fail git cmd 2>err'. Instead, run 'test_must_fail
+ stderr=err git cmd'.
+
On the other hand, don't use test_must_fail for running regular
platform commands; just use '! cmd'. We are not in the business
of verifying that the world given to us sanely works.
@@ -670,6 +674,8 @@ library for your script to use.
Multiple signals can be specified as a comma separated list.
Currently recognized signal names are: sigpipe, success.
(Don't use 'success', use 'test_might_fail' instead.)
+ stderr=<file>:
+ Save <git-command>'s standard error to <file>.
- test_might_fail [<options>] <git-command>
diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 26b149ac1d..d2f561477c 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -618,18 +618,33 @@ list_contains () {
# Multiple signals can be specified as a comma separated list.
# Currently recognized signal names are: sigpipe, success.
# (Don't use 'success', use 'test_might_fail' instead.)
+# stderr=<file>:
+# Save the git command's standard error to <file>.
test_must_fail () {
- case "$1" in
- ok=*)
- _test_ok=${1#ok=}
- shift
- ;;
- *)
- _test_ok=
- ;;
- esac
- "$@"
+ _test_ok= _test_stderr=
+ while test $# -gt 0
+ do
+ case "$1" in
+ stderr=*)
+ _test_stderr=${1#stderr=}
+ shift
+ ;;
+ ok=*)
+ _test_ok=${1#ok=}
+ shift
+ ;;
+ *)
+ break
+ ;;
+ esac
+ done
+ if test -n "$_test_stderr"
+ then
+ "$@" 2>"$_test_stderr"
+ else
+ "$@"
+ fi
exit_code=$?
if test $exit_code -eq 0 && ! list_contains "$_test_ok" success
then
--
2.16.1.180.g07550b0b1b
