Hi,
On Mon, 2023-10-16 at 22:20 +0200, Florian Westphal wrote:
> Thomas Haller <thaller@xxxxxxxxxx> wrote:
> > Passing the test suite must not require latest kernel patches. If
> > test
> > "table_onoff" appears to not work due to a missing kernel patch,
> > skip
> > it.
> >
> > If you run a special kernel and expect that all test pass, set
> > NFT_TEST_FAIL_ON_SKIP=y to catch unexpected skips.
>
> This makes the test suite and all feature probing moot for my use
> cases.
> If I see SKIP, I assume that the feature is missing.
As you probably run a self-built kernel, wouldn't you just `export
NFT_TEST_FAIL_ON_SKIP=y` and reject all skips as failures? What's the
problem with that? That exists exactly for your use case.
The test suite should be usable on some recent + popular distro kernels
(like Fedora 38). That will also be rather important, once the tests
are invoked by `make check` (I have patches for that, it's very simple
and rather nice!).
>
> This is a bug, and it tells me that I might have to do something
> about it.
OK, do you intend to fix this bug in a very timely manner on Fedora 38
(and other popular kernels)? Then maybe hold back the test until that
happend? (or let it skip for now, and in a few weeks, upgrade to hard
failure -- the only problem is not to forget about that).
>
> If you absolutely cannot have a failure because of this, then
> please add another error state for this, so that I can see that
> something is wrong.
>
> This is NOT the same as a skip because some distro kernel lacks
> anonymous chain support.
>
> That said, I would STRONLGY perfer failure here.
> Distros will ship updates that eventually also include this bug fix.
>
> This fix is included in 6.5.6 for example.
>
Ah right. "tests/shell/testcases/transactions/table_onoff" is fixed on
6.5.6-200.fc38.x86_64. There still is a general problem. For example
what about tests/shell/testcases/packetpath/vlan_8021ad_tag ?
Maybe there could be two levels of NFT_TEST_FAIL_ON_SKIP. Maybe
"always" or "bugs-only". Then either:
1) the test would exit 78 instead of 77. And run-test.sh would treat 78
either as failure or as skip, based on NFT_TEST_FAIL_ON_SKIP
2) the test itself could look at NFT_TEST_FAIL_ON_SKIP and decide
whether to exit with 77 or 1.
Or how about adding a mechanism, that compares the kernel version and
decides whether to skip? For example
if ! printf "%s" "$OUT" | grep -q 'counter packets 1 bytes 84' ; then
echo "Filter did not match. Assume kernel lacks fix https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=af84f9e447a65b4b9f79e7e5d69e19039b431c56";
"$NFT_TEST_BASEDIR/helpers/eval-exit-code" kernel 6.5.6 || exit $?
fi
where "eval-exit-code" will either exit with 1 or 77 and log a message
(about the result of the comparison).
The "kernel" arguments indicates to compare the `uname` output in some
useful >= way.
How about "eval-exit-code"?
Thomas
