On 9/25/2018 5:12 PM, Ben Peart wrote:
On 9/25/2018 2:42 PM, Derrick Stolee wrote:
In an effort to ensure new code is reasonably covered by the test
suite, we now have contrib/coverage-diff.sh to combine the gcov
output from 'make coverage-test ; make coverage-report' with the
output from 'git diff A B' to discover _new_ lines of code that are
not covered.
This report takes the output of these results after running on four
branches:
pu: 80e728fa913dc3a1165b6dec9a7afa6052a86325
jch: 0c10634844314ab89666ed0a1c7d36dde7ac9689
next: 76f2f5c1e34c4dbef1029e2984c2892894c444ce
master: fe8321ec057f9231c26c29b364721568e58040f7
master@{1}: 2d3b1c576c85b7f5db1f418907af00ab88e0c303
I ran the test suite on each of these branches on an Ubuntu Linux VM,
and I'm missing some dependencies (like apache, svn, and perforce) so
not all tests are run.
I submit this output without comment. I'm taking a look especially at
my own lines to see where coverage can be improved.
Thanks for driving this. I think it provides an interesting view into
new code and how well it is being tested. In an effort to make this
as useful as possible, we should be looking to eliminate as much noise
as possible otherwise people will stop looking at it.
Thanks for helping identifying the noise.
I looked at the lines that came from my patches and most if not all of
them are only going to be executed by the test suite if the correct
"special setup" option is enabled. In my particular case, that is the
option "GIT_TEST_INDEX_THREADS=<n>" as documented in t/README.
I suspect this will be the case for other code as well so I wonder if
the tests should be run with each the GIT_TEST_* options that exist to
exercise uncommon code paths with the test suite. This should prevent
false positives on code paths that are actually covered by the test
suite as long as it is run with the appropriate option set.
This is a bit tricky to do, but I will investigate. For some things, the
values can conflict with each other (GIT_TEST_SPLIT_INDEX doesn't play
nicely with other index options, I think). For others, we don't have the
environment variables in all versions yet, as they are still merging down.
I realize it would take a long time to run the entire test suite with
all GIT_TEST_* variables so perhaps they can only be tested "as
needed" (ie when patches add new variables in the "special setups"
section of t/README). This should reduce the number of combinations
that need to be run while still eliminating many of the false positive
hits.
This is something to think about. For my own thoughts, I was thinking of
trying to run it when we see large blocks of code that are uncovered and
obviously because of environment variables. This is what I was thinking
when I saw your and Duy's commits in the output. I'll see if I can
re-run the suite using GIT_TEST_INDEX_THREADS=2 and
GIT_TEST_INDEX_VERSION=4.
Thanks,
-Stolee
