Add a mechanism to skip tests. Two cases are implemented:
- skip thest that require JSON support, if nft binary is build without.
- the user can choose to skip long-running tests via the (-Q|--quick) option.
Most tests are fast, so by skipping 8 of 373 tests, I can run the
test suite in 7 seconds.
This is inspired and related to Florian's "feature probing" patchset.
And done with the intent, that those patches could integrate in a
common mechanism. Florian's patches are NOT obsoleted by this
(although, they would require rebase/adjustment).
Changes/notes compared to Florian's approach:
- Florian's patchset mostly called the variable NFT_HAVE_xxx (although,
the commit message also called it NFT_TEST_HAVE_xxx). I choose
NFT_TEST_HAVE_* name to make it clear that the environment variables are
related to the tests.
- The evalution of NFT_TEST_HAVE_json=y|n is hard coded in "run-test.sh"
because it seems special enough. But we certainly can combine that with
the generic mechanism to load features from the "features/" directory.
- NFT_TEST_HAVE_xxx can also be set by the user. That overrides the
feature detection. The point is to force the detection for manual
testing.
- let "test-wrapper.sh" can parse tags like
# NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)
# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
to simplify marking tests with the skip/requires option.
Thomas Haller (2):
tests/shell: skip tests if nft does not support JSON mode
tests/shell: add "--quick" option to skip slow tests (via
NFT_TEST_SKIP_slow=y)
tests/shell/helpers/test-wrapper.sh | 51 ++++++++++++++++-
tests/shell/run-tests.sh | 56 ++++++++++++++++++-
.../shell/testcases/json/0001set_statements_0 | 2 +
tests/shell/testcases/json/0002table_map_0 | 2 +
.../testcases/json/0003json_schema_version_0 | 2 +
.../testcases/json/0004json_schema_version_1 | 2 +
.../shell/testcases/json/0005secmark_objref_0 | 2 +
tests/shell/testcases/json/0006obj_comment_0 | 2 +
tests/shell/testcases/json/netdev | 9 ++-
.../listing/0021ruleset_json_terse_0 | 13 ++++-
.../maps/0004interval_map_create_once_0 | 8 +++
.../testcases/maps/0018map_leak_timeout_0 | 2 +
tests/shell/testcases/maps/vmap_timeout | 2 +
.../testcases/sets/0043concatenated_ranges_0 | 2 +
.../testcases/sets/0044interval_overlap_0 | 2 +
.../testcases/sets/0044interval_overlap_1 | 2 +
tests/shell/testcases/sets/automerge_0 | 2 +
tests/shell/testcases/transactions/0049huge_0 | 14 ++++-
tests/shell/testcases/transactions/30s-stress | 2 +
19 files changed, 168 insertions(+), 9 deletions(-)
--
2.41.0
