Hi Josh, On Wed, 15 Nov 2023, Josh Steadmon wrote: > On 2023.11.13 13:49, Jeff King wrote: > > > > why are the unit tests totally separate from the rest of the suite? I > > would think we'd want them run from one or more t/t*.sh scripts. That > > would make bugs like this impossible, but also: > > > > 1. They'd be run via "make test", so developers don't have to remember > > to run them separately. > > > > 2. They can be run in parallel with all of the other tests when using > > "prove -j", etc. > > The first part is easy, but I don't see a good way to get both shell > tests and unit tests executing under the same `prove` process. For shell > tests, we pass `--exec '$(TEST_SHELL_PATH_SQ)'` to prove, meaning that > we use the specified shell as an interpreter for the test files. That > will not work for unit test executables. Probably my favorite aspect about the new unit tests is that they avoid using the error-prone, unintuitive and slow shell scripts and stay within the programming language of the code that is to be tested: C. > We could bundle all the unit tests into a single shell script, but then > we lose parallelization and add hoops to jump through to determine what > breaks. Or we could autogenerate a corresponding shell script to run > each individual unit test, but that seems gross. Of course, these are > hypothetical concerns for now, since we only have a single unit test at > the moment. I totally agree with you, Josh, that it makes little sense to try to contort the unit tests to be run in the same `prove` run as the regression tests that need to be invoked so totally differently. > There's also the issue that the shell test arguments we pass on from > prove would be shared with the unit tests. That's fine for now, as > t-strbuf doesn't accept any runtime arguments, but it's possible that > either the framework or individual unit tests might grow to need > arguments, and it might not be convenient to stay compatible with the > shell tests. > > Personally, I lean towards keeping things simple and just running a > second `prove` process as part of `make test`. Agreed. > If I was forced to pick a way to get everything under one process, I'd > lean towards autogenerating individual shell script wrappers for each > unit test. But I'm open to discussion, especially if people have other > approaches I haven't thought of. One alternative would be to avoid running the unit tests via `prove` in the first place. For example, we could use the helper from be5d88e11280 (test-tool run-command: learn to run (parts of) the testsuite, 2019-10-04) [*1*]. It would probably need a few improvements, but certainly no wizardry nor witchcraft would be required. It would also help on Windows, where running a simple test helper written in C is vastly faster than running a complex Perl script (which `prove` is). Ciao, Johannes Footnote *1*: I had always wanted to improve that test helper to the point where it could replace our use of `prove`, at least on Windows. It seems, however, that as of 4c2c38e800f3 (ci: modification of main.yml to use cmake for vs-build job, 2020-06-26) we do not use the helper at all anymore. Hopefully it can still be useful. 🤞
