On 01/19/2016 10:45 AM, Emmanuel Dreyfus wrote: > Hi > > Spurious regression failures make developers frustrated. One submits a > change and gets completely unrelated failures. The only way out is to > retrigger regression until it passes, a boring and time-wasting task. > Sometimes after 4 or 5 failed runs, the submitter realize there is a > real issue and look at it, which is a waste of time and resources. > > The fact that we run regression on multiple platforms makes the > situation worse. If you have 10% of chances to hit a spurious failure on > Linux and a 20% chances to hit a spurious failure on NetBSD (random > number chosen), that means you get roughtly a failure for four > submissions (random prediction, as I used random input numbers, but you > get the idea) > > Two solutions are proposed: > > 1) do not run unreliable tests, as proposed by Raghavendra Talur: > http://review.gluster.org/13173 > > I have nothing against the idea, but I voted down the change because it > fails to address the need for different test blacklists on different > platforms: we do not have the same unreliable tests on Linux and NetBSD. > > 2) add a regression option to retry a failed test once, and to validate > the regression if second attempt passes, as I proposed: > http://review.gluster.org/13245 > > The idea is basicaly to automatically do what every submitter has been > doing: retry without a thought when regression fails. The benefit of > this approach is also that it gives us a better view of what test failed > because of the change, and what test failed because it was unreliable. > > The retry feature is optionnal and triggered by using the -r flag to > run-tests.sh. I intend to use it on NetBSD regression to reduce the > number of failures that annoy people. It could be used on Linux > regression too, though I do not plan to touch that on my own. +1 to option 2 > > Please people tell us what approach you prefer. > _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
