Hi Peff, On Tue, 5 Mar 2019, Jeff King wrote: > On Tue, Mar 05, 2019 at 07:04:59PM +0700, Duy Nguyen wrote: > > > On Mon, Feb 4, 2019 at 4:17 PM Christian Couder > > <christian.couder@xxxxxxxxx> wrote: > > > > > > Hi everyone, > > > > > > There are now ideas, micro-projects and organization application pages > > > for GSoC 2019 on https://git.github.io/ > > > > > > It would be nice to have a few more project ideas. > > > > Not sure if it's too late now. Anyway this could be something fun to > > do: support C-based tests in our test suite. > > > > A while back I noticed some test running very long because it was > > trying a lot of input combination. The actual logic is not much, but > > because of the increasing number of test cases, overhead goes off the > > roof. The last part is probably not true, but Windows port I think is > > hit much harder than what I experience, and I think Dscho did complain > > about it. > > > > So what this project does is somehow allow people to write test cases > > in C instead of shell. Imagine replacing t3070-wildmatch.sh with a > > binary program t3070-wildmatch that behaves the same way. This test > > framework needs to support the same basic feature set as test-lib.sh: > > TAP output, test results summary, maybe -i and --valgrind... To > > demonstrate that the test framework works, one of these long test > > files should be rewritten in C. I'm sure there's one that is simple to > > rewrite. > > > > I'm pretty sure I had some fun with this idea and made some prototype > > but I couldn't find it. If I do, I'll post the link here. > > In my experience, it's nicer to have a tool written in C that can be > driven by arbitrary input. That makes it easy to write new test cases, > because you just have to write in some easy domain-specific format > instead of embedding the test data in C code. > > And many of our tests do work like that (in fact, many of the Git > plumbing tools function as that). E.g., test-date gives you direct > access to the low-level routines, and we feed it a variety of dates. > > That doesn't help with the cost of invoking that tool over and over, > though, once per test case. I wonder if we could have some kind of > hybrid. I.e., where t3070 is still a shell script, but it primarily > consists of running one big binary, like: > > test-wildmatch <<-\EOF > case 1 > case 2 > ...etc > EOF > > but with one added twist: test-wildmatch would actually generate TAP > output for each test, rather than just returning 0/1 for each success or > failure, and being embedded in a test_expect_success. > > It seems like that would even be pretty easy to do, with the exception > of the numbering. It would be nice if we could intermingle this kind of > "chunk of C tests" with normal tests, but we'd have to figure out how > many tests it ran and increment our shell-script's counter > appropriately. Oooooh, that sounds like a very nice idea! Eventually, we might even be able to specify our test cases in our own, extensible language, where we do not have to pay attention to &&-chains, or portability, because our test runner does all that for us, under the hood, as it should be. Dreaming of that future, Dscho
