David Gilbert wrote: > 1) constant number of tests, regardless of exceptions being thrown or > Mauve does have a design flaw where it can be tricky to automatically > assign a unique identifier to each check(), and this makes it hard to > compare two Mauve runs (say a test of the latest Classpath CVS vs the > last release, or the Classpath vs JDK 1.5 - both of which would be > interesting). Right. We all understand the problem - its just the solution is what we need to agree on :) > I think the absolute number is meaningless however you count the > tests, so I don't see this as an advantage. Yes, numbers alone are meaningless, but with the current design, all the results are meaningless without a lot of context. The real issue is having a simple way to uniquely identify each test case for the purpose of identifying regressions. This becomes fundamentally much easier when 1 test() method corresponds to one test case. It is not reasonable to expect test case developers ensure that all tests "run linearly". Exceptions can potentially be thrown at any time, so to ensure linearity, every check() call would need wrapped with a try/catch. > You'll lose the ability to distinguish between an existing failure > where (say) 1 out of 72 checks fail, and after some clever patch 43 > out of 72 checks fail, but the new system reports both as 1 test failure. This is a valid concern. However, we will still track exactly which check() calls fail so that in the event of a test failure, a full diagnostic can be provided. In addition, we can still count the total number of check() calls executed, for statistical purposes. If the reduced test-case granularity does prove to be problematic in some cases - say some test() where a small number of checks fail for cases that are "hard to fix" problems and thus should be "xfailed" (to use gcc/dejagnu lingo), then that test case should probably be split. Alternatively, to avoid splitting things across multiple test classes, we could add JUnit-style support for multiple test() methods in a single test class. Bryce
