On 9/26/2018 6:56 AM, Jason Pyeron wrote:
-----Original Message-----
From: Derrick Stolee
Sent: Wednesday, September 26, 2018 6:43 AM
On 9/25/2018 5:12 PM, Ben Peart wrote:
On 9/25/2018 2:42 PM, Derrick Stolee wrote:
In an effort to ensure new code is reasonably covered by the test
suite, we now have contrib/coverage-diff.sh to combine the gcov
output from 'make coverage-test ; make coverage-report' with the
output from 'git diff A B' to discover _new_ lines of code that are
not covered.
<snip/>
I looked at the lines that came from my patches and most if not all of
them are only going to be executed by the test suite if the correct
"special setup" option is enabled. In my particular case, that is the
option "GIT_TEST_INDEX_THREADS=<n>" as documented in t/README.
I suspect this will be the case for other code as well so I wonder if
the tests should be run with each the GIT_TEST_* options that exist to
exercise uncommon code paths with the test suite. This should prevent
false positives on code paths that are actually covered by the test
suite as long as it is run with the appropriate option set.
This is a bit tricky to do, but I will investigate. For some things, the
values can conflict with each other (GIT_TEST_SPLIT_INDEX doesn't play
nicely with other index options, I think). For others, we don't have the
environment variables in all versions yet, as they are still merging down.
Remember that the code coverage is cumulative, on one of my projects where I have similar special cases the automated build will run through a sequence of different build options (including architectures) and executions - then generate the final report.
For Git, I'm just using "make coverage-test" and "make coverage-report".
To make the results cumulative, I'd need to remove the
"coverage-clean-results" as a dependency to "coverage-test". This may be
a good thing to do, so I can run the test suite with multiple options.
Perhaps I'll just add a new "coverage-test-with-options" step that just
runs the tests in "default" mode and then "enhanced" mode.
Another thought would be to weight the report for "new lines of code" - which is the same we do for our Fortify scans. We take the git blame for the change range and de-prioritize the findings for lines outside of the changed lines. This could give a tighter focus on the newly change code's test coverage.
This is what the contrib/coverage-diff.sh script does. I'd love your
input, as it is still under review [1].
[1]
https://public-inbox.org/git/21214cc321f80cf2e9eb0cdb1ec3ebb869ea496d.1537542952.git.gitgitgadget@xxxxxxxxx/
[PATCH v3 1/1] contrib: add coverage-diff script
