Hi,
On Tue, May 31, 2022 at 2:07 PM Stephan Bergmann <sbergman@xxxxxxxxxx> wrote:
So the general advice would be to ignore occasional failed builds (which
might not only fail due to spurious test failures, but also because e.g.
a build breaker got submitted by accident). If some specific tests
cause enough builds to fail to make that approach impractical, those
tests should get fixed. Or, as a last resort, get disabled for
known-failing build scenarios.
I don't think -k would be a good solution, as it would make it harder to
meaningfully interpret the generated data.
Ok, so let me see if I understood that correctly :
*if* lcov reports were to be generated automatically on a regular basis again, then the desired behaviour would be to just let the build fail/stop (whatever the reason, gcov related, spuriously failure, developer mistake), don't generate a lcov report when it does fail, and if it fails often enough within a certain timeframe people will investigate and determine whether the failing check should be fixed or disabled, and a new gcov/lcov report should only be generated if the build and 'make check' both succeed.
*if* lcov reports were to be generated automatically on a regular basis again, then the desired behaviour would be to just let the build fail/stop (whatever the reason, gcov related, spuriously failure, developer mistake), don't generate a lcov report when it does fail, and if it fails often enough within a certain timeframe people will investigate and determine whether the failing check should be fixed or disabled, and a new gcov/lcov report should only be generated if the build and 'make check' both succeed.
Occasionally failing tests are a well-known problem for LO (e.g.,
witness the "Jenkins / CI update: tests that failed more than twice in
last seven days" section in the weekly ESC minutes---many of those are
apparently spuriously failing tests), and there is no reason to assume
that your gcov builds would not also occasionally be affected by that.
Although I realize - as you explained - that there are occasional test failures for all Jenkins builds, I am also not sure if this is what is going on here specifically. In order to better determine if we are dealing with occasional or structurally failing tests for the mentioned 3 tests specifically (UITest_solver, CppunitTest_sccomp_solver, CppunitTest_sccomp_swarmsolvertest), I did some more testing (fresh git pull and build), and when I do a gcov build and make check, these 3 fail for me reliably (and they succeed for a non-gcov 'regular' build). All the other tests succeed, but these 3 keep failing for me. I ran them multiple times in a row, and they keep failing for each run. So while I realise that this might not be 100% proof, I also get the strong impression that for these 3 specifically, we are not talking about 'occasionally' failing tests, but about structurally failing tests when run on a gcov build, and this needs to be looked into. Of course, the fact that they fail reliably for me on every run I tried says nothing about what makes that so: for example, it can still be timeout related, as I can imagine (but did not test) a gcov enabled test taking longer to complete than a test run on a 'regular' build. The reason that I keep going on about this is not because I am purposefully trying to be annoying, but rather because I personally strongly think/feel (but I may be mistaken) that in order for people to be interested in reviving automated lcov reports at all, it first needs to be made sure that the automation works 'most of the time' (or at least as much as the other Jenkins builds), and not fail on every single run right at the start.
- Maarten
