Hi Eric, On Wed, 5 Sep 2018, Eric Sunshine wrote: > On Wed, Sep 5, 2018 at 8:39 AM Johannes Schindelin > <Johannes.Schindelin@xxxxxx> wrote: > > So let's hear some ideas how to improve the situation, m'kay? Just as > > a reminder, this is the problem I want to solve: I want to run the > > tests in a light-weight manner, with minimal output, and only in case > > of an error do I want to crank up the verbosity. Instead of wasting > > most of the effort to log everything and then throwing it away in most > > of the common cases, I suggest to re-run the entire test. > > What about the very different approach of capturing the full "verbose" > output the executed tests in addition to whatever is actually output > to the terminal? I fear it is not really possible to do a "verbose but not really" mode. I want the console output to be quiet, there is no use in chatting up the build log with these messages, as we have to run the tests in parallel, so the output would be utterly hard to interpret anyway. At the same time, I want verbose output for use in the test results. It is not really possible to `tee` all output, then "quiet down" the part that makes it into the log. > If a test fails, then (and only then) you can insert the captured > verbose output into the JUnit XML file. Yep. That's what my patch series does. If a test fails, then (and only then) I re-run the script with verbose output that is immediately moved into that JUnit XML file. > This way (if we always have the full verbose output at hand), you don't > need to re-run the test at all. But that way, if we always have the full verbose output, the build log will be unnecessarily verbose and confusing. > I've cc:'d Peff and Jonathan since I believe they are more familiar > with how all the capturing / output-redirection works in the test > suite. Okay. Ciao, Dscho P.S.: An unintended side effect of re-running the tests is to identify flakey tests. I do not yet have a way to represent this outcome in the test result, but I deem this an additional benefit in favor of keeping my current strategy.
