On Tue, Mar 05, 2019 at 07:04:59PM +0700, Duy Nguyen wrote: > On Mon, Feb 4, 2019 at 4:17 PM Christian Couder > <christian.couder@xxxxxxxxx> wrote: > > > > Hi everyone, > > > > There are now ideas, micro-projects and organization application pages > > for GSoC 2019 on https://git.github.io/ > > > > It would be nice to have a few more project ideas. > > Not sure if it's too late now. Anyway this could be something fun to > do: support C-based tests in our test suite. > > A while back I noticed some test running very long because it was > trying a lot of input combination. The actual logic is not much, but > because of the increasing number of test cases, overhead goes off the > roof. The last part is probably not true, but Windows port I think is > hit much harder than what I experience, and I think Dscho did complain > about it. > > So what this project does is somehow allow people to write test cases > in C instead of shell. Imagine replacing t3070-wildmatch.sh with a > binary program t3070-wildmatch that behaves the same way. This test > framework needs to support the same basic feature set as test-lib.sh: > TAP output, test results summary, maybe -i and --valgrind... To > demonstrate that the test framework works, one of these long test > files should be rewritten in C. I'm sure there's one that is simple to > rewrite. > > I'm pretty sure I had some fun with this idea and made some prototype > but I couldn't find it. If I do, I'll post the link here. In my experience, it's nicer to have a tool written in C that can be driven by arbitrary input. That makes it easy to write new test cases, because you just have to write in some easy domain-specific format instead of embedding the test data in C code. And many of our tests do work like that (in fact, many of the Git plumbing tools function as that). E.g., test-date gives you direct access to the low-level routines, and we feed it a variety of dates. That doesn't help with the cost of invoking that tool over and over, though, once per test case. I wonder if we could have some kind of hybrid. I.e., where t3070 is still a shell script, but it primarily consists of running one big binary, like: test-wildmatch <<-\EOF case 1 case 2 ...etc EOF but with one added twist: test-wildmatch would actually generate TAP output for each test, rather than just returning 0/1 for each success or failure, and being embedded in a test_expect_success. It seems like that would even be pretty easy to do, with the exception of the numbering. It would be nice if we could intermingle this kind of "chunk of C tests" with normal tests, but we'd have to figure out how many tests it ran and increment our shell-script's counter appropriately. -Peff
