On 31/05/2022 13:43, Maarten Hoes wrote:
Anyway, if more tests fail sometimes without there being a 'real' failure, then I am not sure how to deal with that in relation to generating an lcov report for a full build. My initial idea was to not generate a report if 'make check' failed, but hearing this now makes me wonder if that would be a good approach. Perhaps it would be preferred just to run 'make -k check' and generate a report always, regardless of test failures ? Of course you could also choose to skip such tests, but that would lead to not representative results; and someone would have to manually keep the skip list up to date, which people will forget to do.
Occasionally failing tests are a well-known problem for LO (e.g., witness the "Jenkins / CI update: tests that failed more than twice in last seven days" section in the weekly ESC minutes---many of those are apparently spuriously failing tests), and there is no reason to assume that your gcov builds would not also occasionally be affected by that.
So the general advice would be to ignore occasional failed builds (which might not only fail due to spurious test failures, but also because e.g. a build breaker got submitted by accident). If some specific tests cause enough builds to fail to make that approach impractical, those tests should get fixed. Or, as a last resort, get disabled for known-failing build scenarios.
I don't think -k would be a good solution, as it would make it harder to meaningfully interpret the generated data.
