> -----Original Message----- > From: Derrick Stolee > Sent: Wednesday, September 26, 2018 6:43 AM > > On 9/25/2018 5:12 PM, Ben Peart wrote: > > > > > > On 9/25/2018 2:42 PM, Derrick Stolee wrote: > >> In an effort to ensure new code is reasonably covered by the test > >> suite, we now have contrib/coverage-diff.sh to combine the gcov > >> output from 'make coverage-test ; make coverage-report' with the > >> output from 'git diff A B' to discover _new_ lines of code that are > >> not covered. > >> <snip/> > > > > I looked at the lines that came from my patches and most if not all of > > them are only going to be executed by the test suite if the correct > > "special setup" option is enabled. In my particular case, that is the > > option "GIT_TEST_INDEX_THREADS=<n>" as documented in t/README. > > > > I suspect this will be the case for other code as well so I wonder if > > the tests should be run with each the GIT_TEST_* options that exist to > > exercise uncommon code paths with the test suite. This should prevent > > false positives on code paths that are actually covered by the test > > suite as long as it is run with the appropriate option set. > This is a bit tricky to do, but I will investigate. For some things, the > values can conflict with each other (GIT_TEST_SPLIT_INDEX doesn't play > nicely with other index options, I think). For others, we don't have the > environment variables in all versions yet, as they are still merging down. Remember that the code coverage is cumulative, on one of my projects where I have similar special cases the automated build will run through a sequence of different build options (including architectures) and executions - then generate the final report. Another thought would be to weight the report for "new lines of code" - which is the same we do for our Fortify scans. We take the git blame for the change range and de-prioritize the findings for lines outside of the changed lines. This could give a tighter focus on the newly change code's test coverage. > > > > I realize it would take a long time to run the entire test suite with > > all GIT_TEST_* variables so perhaps they can only be tested "as > > needed" (ie when patches add new variables in the "special setups" > > section of t/README). This should reduce the number of combinations > > that need to be run while still eliminating many of the false positive > > hits. > > This is something to think about. For my own thoughts, I was thinking of > trying to run it when we see large blocks of code that are uncovered and > obviously because of environment variables. This is what I was thinking > when I saw your and Duy's commits in the output. I'll see if I can > re-run the suite using GIT_TEST_INDEX_THREADS=2 and > GIT_TEST_INDEX_VERSION=4. > > Thanks, > -Stolee
