Jakub Kicinski wrote: > On Wed, 28 Aug 2024 15:33:33 -0400 Willem de Bruijn wrote: > > That could work. > > > > Is reporting one KTAP and exitcode per directory vs per packetdrill > > invocation good/bad/neither? > > To me "neither", hopefully and unhelpfully I won't tell you anything > that's not common sense :) The balance is between: > > - having test cases which don't take too long (assuming debug kernel): > <15min is good, > >1h I will start complaining, > >1h30m is bad because we can't retry and still make the 3h > deadline that NIPA has, > >3h the test can't run in NIPA at all. > > vs > > - flip side is having so many cases it's hard to keep track and render > in the UI. JSON is relatively slow to process. If you have 150 cases, > that's 300 per branch (debug and non-debug kernels), times 8 > branches a day => 2.4k results / day > I think that's still fine-ish, but on the larger side for sure. For > reference net and forwarding have ~100 tests each. FWIW we do have > the ability to collect and display nested KTAP so the information is > not lost (but it makes queries slower so we don't fetch it by default). > > > Three other issues if this is calling packetdrill directly is > > - passing the non-trivial IP specific flags > > - running twice, for IPv4 and IPv6 > > - chdir into the directory of the pkt file > > > > That can be addressed by instead calling a small wrapper shell script. > > > > That would do the test_func_builder part of packetdrill_ksft.py. > > But without the need to handle netns, popen/cmd, etc, and thus the > > ksft dependencies. > > Right! Implemented this, but hit a snag Kselftest install does not preserve directories. So all .pkt files are copied into net/packetdrill root. This is messy. More fundamentally it breaks the includes in the files (e..g, `source ../common/defaults.sh`). The output of having each test separate is also quite unreadable. An alternative is to add each subdirectory as a separate TARGET. But that seems similarly impractical. And does nothing to group KTAP output. In practice, most directories have a handful of .pkt scripts, and each script is fast, so serialization is not a huge issue. I found tools/testing/selftests/net/kselftest/ktap_helpers.sh, which has helpful KTAP boilerplate for shell tests, including KSFT exit codes. Am using that instead of python, to reduce the external dependencies.
