On 06/13/2017 02:22 PM, Jesper Dangaard Brouer wrote:
On Tue, 13 Jun 2017 13:09:26 +0200
Daniel Borkmann <borkmann@xxxxxxxxxxxxx> wrote:
On 06/13/2017 12:30 PM, Jesper Dangaard Brouer wrote:
Hi Fengguang and Daniel,
I'm looking into adding some tests for tools/testing/selftests/bpf/.
I'm a bit confused about the output format, as it seems to differ even
among programs in the same directory (tools/testing/selftests/bpf/).
I do see, that tools/testing/selftests/lib.mk have a standard output
format via the program shell exit-code (copied Makefile syntax):
exit_code && echo "selftests: $$BASENAME_TEST [PASS]" \
|| echo "selftests: $$BASENAME_TEST [FAIL]"
To Fengguang:
How do your kbuild-robot hook into this?
(1) Do you parse/grep after output? (egrep -e "^selftests: .* \[.*\]")
(2) Or do you rely on the shell exit codes?
A quick look at some of the bpf tests, show that they are not all using
the exit codes to indicate failure (... which I guess is a bug we
should fix?)
Yep, if you have a chance please send a fix (otherwise I can also take
care of it). The way test_verifier handles it wrt exit code is correct,
and other tests should follow the same model. Thanks for spotting!
Sure, I can fix it... I know you are a busy man :-)
Looking at the code for test_verfier.c, I don't agree that it is using
return codes correctly. The shell exit code should not be a negative
number (you return -errors).
/usr/include/stdlib.h
#define EXIT_FAILURE 1 /* Failing exit status. */
#define EXIT_SUCCESS 0 /* Successful exit status. */
Further more the exit code is masked with 255, meaning that if you get
256 errors, and return -256 masked it will be 0, which would indicate
success.
Lets hope we don't ever break half the verifier. :) Yeah, agree, this
also needs to be fixed.
My suggestion would be to add a small helper to selftests/bpf/bpf_util.h
along the lines of:
static inline int bpf_report(int passes, int errors)
{
printf("Summary: %d PASSED, %d FAILED\n", passes, errors);
return errors ? EXIT_FAILURE : EXIT_SUCCESS;
}
Which can then be used by *all* BPF tests on exit, so we can make this
more consistent wrt exit code and summary line.
Could perhaps even be used outside of BPF later on, but up to Shuah.
p.s. I have noticed: Documentation/kselftest.txt, but it says nothing
about output format used/required program exit codes.
--
To unsubscribe from this list: send the line "unsubscribe linux-kselftest" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
