On Tue, Jun 13, 2017 at 12:30:55PM +0200, Jesper Dangaard Brouer wrote:
Hi Fengguang and Daniel,
I'm looking into adding some tests for tools/testing/selftests/bpf/.
I'm a bit confused about the output format, as it seems to differ even
among programs in the same directory (tools/testing/selftests/bpf/).
We were kind of confused, too. ;)
The diversity output resulted in a long parse script (listed below).
I do see, that tools/testing/selftests/lib.mk have a standard output
format via the program shell exit-code (copied Makefile syntax):
exit_code && echo "selftests: $$BASENAME_TEST [PASS]" \
|| echo "selftests: $$BASENAME_TEST [FAIL]"
To Fengguang:
How do your kbuild-robot hook into this?
(1) Do you parse/grep after output? (egrep -e "^selftests: .* \[.*\]")
(2) Or do you rely on the shell exit codes?
It's mainly (1). The parse script is here:
https://github.com/01org/lkp-tests/blob/master/stats/kernel_selftests
Here is how we invoke the tests:
https://github.com/01org/lkp-tests/blob/master/tests/kernel_selftests
The infrastructure catches exit code changes of the above script.
However that script ignores the individual exit code of `make`. The
parse script does catch "recipe for target.+failed" errors and turn
them into "#{testname}.make_fail: 1" stats.
A quick look at some of the bpf tests, show that they are not all using
the exit codes to indicate failure (... which I guess is a bug we
should fix?)
p.s. I have noticed: Documentation/kselftest.txt, but it says nothing
about output format used/required program exit codes.
That could be trouble for auto test runners. It'd be great if there is
a standard on exit codes and a set of common output patterns that the
output parser can rely on. So that when new tests are added, the
output parsing will just work in various test infrastructures.
Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
