On Mon, 4 Mar 2024 22:20:03 +0000 Mark Brown wrote:
On Wed, Feb 28, 2024 at 04:59:07PM -0800, Jakub Kicinski wrote:
When running selftests for our subsystem in our CI we'd like all
tests to pass. Currently some tests use SKIP for cases they
expect to fail, because the kselftest_harness limits the return
codes to pass/fail/skip. XFAIL which would be a great match
here cannot be used.
Remove the no_print handling and use vfork() to run the test in
a different process than the setup. This way we don't need to
pass "failing step" via the exit code. Further clean up the exit
codes so that we can use all KSFT_* values. Rewrite the result
printing to make handling XFAIL/XPASS easier. Support tests
declaring combinations of fixture + variant they expect to fail.
This series landed in -next today and has caused breakage on all
platforms in the ALSA pcmtest-driver test. When run on systems that
don't have the driver it needs loaded the test skip but since this
series was merged skipped tests are logged but then reported back as
failures:
# selftests: alsa: test-pcmtest-driver
# TAP version 13
# 1..5
# # Starting 5 tests from 1 test cases.
# # RUN pcmtest.playback ...
# # SKIP Can't read patterns. Probably, module isn't loaded
# # playback: Test failed
# # FAIL pcmtest.playback
# not ok 1 pcmtest.playback # Can't read patterns. Probably, module isn't loaded
# # RUN pcmtest.capture ...
# # SKIP Can't read patterns. Probably, module isn't loaded
# # capture: Test failed
# # FAIL pcmtest.capture
# not ok 2 pcmtest.capture # Can't read patterns. Probably, module isn't loaded
# # RUN pcmtest.ni_capture ...
# # SKIP Can't read patterns. Probably, module isn't loaded
# # ni_capture: Test failed
# # FAIL pcmtest.ni_capture
# not ok 3 pcmtest.ni_capture # Can't read patterns. Probably, module isn't loaded
# # RUN pcmtest.ni_playback ...
# # SKIP Can't read patterns. Probably, module isn't loaded
# # ni_playback: Test failed
# # FAIL pcmtest.ni_playback
# not ok 4 pcmtest.ni_playback # Can't read patterns. Probably, module isn't loaded
# # RUN pcmtest.reset_ioctl ...
# # SKIP Can't read patterns. Probably, module isn't loaded
# # reset_ioctl: Test failed
# # FAIL pcmtest.reset_ioctl
# not ok 5 pcmtest.reset_ioctl # Can't read patterns. Probably, module isn't loaded
# # FAILED: 0 / 5 tests passed.
# # Totals: pass:0 fail:5 xfail:0 xpass:0 skip:0 error:0
I haven't completely isolated the issue due to some other breakage
that's making it harder that it should be to test.
A sample full log can be seen at:
https://lava.sirena.org.uk/scheduler/job/659576#L1349
Thanks! the exit() inside the skip evaded my grep, I'm testing this:
diff --git a/tools/testing/selftests/alsa/test-pcmtest-driver.c b/tools/testing/selftests/alsa/test-pcmtest-driver.c
index a52ecd43dbe3..7ab81d6f9e05 100644
--- a/tools/testing/selftests/alsa/test-pcmtest-driver.c
+++ b/tools/testing/selftests/alsa/test-pcmtest-driver.c
@@ -127,11 +127,11 @@ FIXTURE_SETUP(pcmtest) {
int err;
if (geteuid())
- SKIP(exit(-1), "This test needs root to run!");
+ SKIP(exit(KSFT_SKIP), "This test needs root to run!");
err = read_patterns();
if (err)
- SKIP(exit(-1), "Can't read patterns. Probably, module isn't loaded");
+ SKIP(exit(KSFT_SKIP), "Can't read patterns. Probably, module isn't loaded");
card_name = malloc(127);
ASSERT_NE(card_name, NULL);
diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c
index 20294553a5dd..356ba5f3b68c 100644
--- a/tools/testing/selftests/mm/hmm-tests.c
+++ b/tools/testing/selftests/mm/hmm-tests.c
@@ -138,7 +138,7 @@ FIXTURE_SETUP(hmm)
self->fd = hmm_open(variant->device_number);
if (self->fd < 0 && hmm_is_coherent_type(variant->device_number))
- SKIP(exit(0), "DEVICE_COHERENT not available");
+ SKIP(exit(KSFT_SKIP), "DEVICE_COHERENT not available");
ASSERT_GE(self->fd, 0);
}
@@ -149,7 +149,7 @@ FIXTURE_SETUP(hmm2)
self->fd0 = hmm_open(variant->device_number0);
if (self->fd0 < 0 && hmm_is_coherent_type(variant->device_number0))
- SKIP(exit(0), "DEVICE_COHERENT not available");
+ SKIP(exit(KSFT_SKIP), "DEVICE_COHERENT not available");
ASSERT_GE(self->fd0, 0);
self->fd1 = hmm_open(variant->device_number1);
ASSERT_GE(self->fd1, 0);
but there's no more context. I'm also seeing some breakage in the
seccomp selftests which also use kselftest-harness:
# # RUN TRAP.dfl ...
# # dfl: Test exited normally instead of by signal (code: 0)
# # FAIL TRAP.dfl
# not ok 56 TRAP.dfl
# # RUN TRAP.ign ...
# # ign: Test exited normally instead of by signal (code: 0)
# # FAIL TRAP.ign
# not ok 57 TRAP.ign
Ugh, I'm guessing vfork() "eats" the signal, IOW grandchild signals,
child exits? vfork() and signals.. I'd rather leave to Kees || Mickael.
