The current "make check" job on pull requests is configured to require
a "passing/OK" state to allow a merge.
Looking back at the past 100 builds since March 13th, there is roughly a 20%
failure rate [0]. This is a similar failure rate for ceph-volume PRs which never
hit any make check paths: 6 failures out of the last 25 ceph-volume
pull requests have
make check failures).
These failures in make check means that we must almost always ignore them, and
use administrator privilege to merge. This is far from ideal, and further
reduces the confidence in the tests.
Some of the failures are produced by code that implies a grey area, enough to
do a non-zero exit status:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/test-map-pgs.t:
failed
--- /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/test-map-pgs.t
+++ /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/osdmaptool/test-map-pgs.t.err
@@ -40,6 +40,7 @@
# it is almost impossible to get the same stats with random and crush
# if they are, it most probably means something went wrong somewhere
$ test "$STATS_CRUSH" != "$STATS_RANDOM"
+ [1]
# Ran 13 tests, 0 skipped, 1 failed.
Without a doubt, we will hit environmental build issues, and re-triggering is
fine, but then again, why are we having a *mandatory check* that has a high
failure rate from incorrect assumptions?
We've discussed the possibility of "gating" pull requests, and the make check
tool has been suggested for this. I would love to see that happen at some
point, but that will take significant effort on make check, or a separate tool
that can start with higher-confidence checks.
[0] https://jenkins.ceph.com/job/ceph-pull-requests/buildTimeTrend
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
