On Wed, Aug 22, 2018 at 7:53 PM Eric Sunshine <sunshine@xxxxxxxxxxxxxx> wrote:
> Can you say a word or two (here in the email thread) about how you're
> finding these failures (across the various test fixes you've posted
> recently)? Are you instrumenting the code in some fashion? Or, finding
> them by visual inspection?
Errors from system commands in our tests look like these:
grep: file3: No such file or directory
sed: -e expression #1, char 2: extra characters after command
diff: sub1/.git: No such file or directory
tar: rmtlseek not stopped at a record boundary
tar: Error is not recoverable: exiting now
while errors from the shell running the test like these:
t0020-crlf.sh: 8: eval: cannot open two: No such file
t6018-rev-list-glob.sh: 4: eval: cannot open expect: No such file
t7408-submodule-reference.sh: 615: test: =: unexpected operator
i.e. lines starting with various system commands' or test scripts'
names, followed by ': '.
So I've modified t/Makefile to not remove the 'test-results' directory
after a successful 'make test':
diff --git a/t/Makefile b/t/Makefile
index ea36cf7ac7..c7b1655593 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -54,10 +54,11 @@ pre-clean:
$(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'
clean-except-prove-cache:
- $(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
+ $(RM) -r 'trash directory'.*
$(RM) -r valgrind/bin
clean: clean-except-prove-cache
+ $(RM) '$(TEST_RESULTS_DIRECTORY_SQ)'
distclean: clean
$(RM) .prove
And then scanned the results of a '--verbose-log -x' test run with:
grep -E '^(awk|basename|cat|cd|chmod|cmp|cp|cut|diff|dirname|egrep|find|fgrep|grep|gunzip|gzip|ln|mkdir|mkfifo|mktemp|mv|readlink|rmdir|sed|sort|tar|touch|tr|ulimit|umask|uniq|unzip|wc|zipinfo|t[0-9][0-9][0-9][0-9]-[^:]*\.sh):
' test-results/*.out
and then, for lack of something better to do ;), I started looking at
the simpler looking errors.
I've though about how a check like this could be automated, but
haven't had any workable idea yet. There are commands that can
legitimately print errors, e.g. when checking for a prereq which the
system doesn't have (e.g. the 'tar' errors above, I think). And the
list of system commands in the grep pattern above is surely incomplete
and will likely change in the future...
