Today was a bad day. Mostly it was my fault, because there were two problems with the patch I submitted to make tests fail more quickly. It was incorrectly reporting failed tests as successful. It was causing regression to complete before smoke most of the time, which exposed a bug in how we handle the Verified flag in regression and smoke. Specifically, smoke would *clear* the flag already set by regression, allowing the netbsd-regression result to dominate the overall result. Either problem would cause a patch to show up as V+1 when it shouldn't have. As penance, I spent much of the day (normally a day off BTW) manually checking results and resetting these flags. If your patch got a result of "FAILURE" (not "FAILED") or "FAILED (script error)" that's what happened. To be clear: the result is exactly what it would have been before, only arrived at more quickly even with the human delay. I'd be interested in feedback on how that's working out for people. After all of this, it became apparent that this was one of our worst regression-testing days so far. It's easier to look on Gerrit than on Jenkins because of all the false positives, and right now I see only *eight* of the most recent fifty patches passed regression. Two of those don't count, because they're my own changes to the test scripts (more about those in a moment) and three more have no result at all yet. The most common cause of failure was volume-snapshot-clone.t, with uss.t coming in a fairly distant second . . . which brings me to those test script changes. There are actually two on master, with a backport of one for 3.7: http://review.gluster.org/10259 http://review.gluster.org/10255 (backport to 3.7) http://review.gluster.org/10277 The last one might be controversial, because it explicitly skips the two tests mentioned above. *Skipping regression tests is a shameful thing*! It frustrates me no end that things have gotten this bad, and we absolutely should fix those tests before we release anything, but meanwhile they're getting in the way of doing any *other* work. We need to get the tests for that other work flowing, and *simultaneously* work on fixing the failing regression tests from these two on down. Yes, I know I'm shouting. Sorry about that. I just want to get us out of this abyss before I hit the road for Bangalore and Barcelona in the next few weeks. Lastly, I have a lead on some of the core dumps that have occurred during regression tests. See the following bug for details. https://bugzilla.redhat.com/show_bug.cgi?id=1212660 _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
