>>>>> "David" == David Gilbert <david.gilbert@xxxxxxxxxxxxxxxxxxx> writes: >> Another suggestion that Tom Fitzsimmons had was to change the way we >> count the number of tests. Counting each invocation of the test() >> method rather than each call to harness.check() has two benefits: David> We can work around that by ensuring that all the tests run linearly David> (no if-else branches - I've written a large number of tests this way David> and not found it to be a limitation, but I don't know what lurks in David> the depths of the older Mauve tests). There is still the problem that David> an exception being thrown during a test means some checks don't get David> run, but a new Mauve comparison report (not yet developed, although David> I've done a little experimenting with it) could highlight those. I've always tried to write tests the way you suggest, but the exception problem turns out to be a real one, preventing test stability in some cases. One thing I like about this current proposal is that it automates test stability -- the only failure modes possible are if a test hangs or if the VM crashes. As far as having more granular information -- we can still print a message when a check() fails. A command line option to the test harness could control this, for instance. I think we don't want to just print a plain 'FAIL', we want some explanation; the detailed info could go there. Tom
