Thomas Haller <thaller@xxxxxxxxxx> wrote: > As you probably run a self-built kernel, wouldn't you just `export > NFT_TEST_FAIL_ON_SKIP=y` and reject all skips as failures? What's the > problem with that? That exists exactly for your use case. No, its not my use case. The use case is to make sure that the frankenkernels that I am in charge of do not miss any important bug fixes. This is the reason for the feature probing, "skip" should tell me that I can safely ignore it because the feature is not present. I could built a list of "expected failures", but that will mask real actual regressions. > > This is a bug, and it tells me that I might have to do something > > about it. > > OK, do you intend to fix this bug in a very timely manner on Fedora 38 > (and other popular kernels)? Then maybe hold back the test until that > happend? (or let it skip for now, and in a few weeks, upgrade to hard > failure -- the only problem is not to forget about that). I did keep the test back until I saw that -stable had picked it up. I can wait longer, sure. > Ah right. "tests/shell/testcases/transactions/table_onoff" is fixed on > 6.5.6-200.fc38.x86_64. There still is a general problem. For example > what about tests/shell/testcases/packetpath/vlan_8021ad_tag ? Its also a bug that needs to be fixed in the kernel. I applied it after stable had picked it up for 6.5.7. > 1) the test would exit 78 instead of 77. And run-test.sh would treat 78 > either as failure or as skip, based on NFT_TEST_FAIL_ON_SKIP > > 2) the test itself could look at NFT_TEST_FAIL_ON_SKIP and decide > whether to exit with 77 or 1. > > > Or how about adding a mechanism, that compares the kernel version and > decides whether to skip? For example I don't think that kernel versions work or are something that we can realistically handle. Even just RHEL would be a nightmare if one considers all the different release streams. I think even just handling upstream -stable is too much work. That said, I hope that these kinds of tests will happen less frequently over time.
