On Thu, 2020-09-03 at 11:15 +0100, Andrew Price wrote: > Hi, > > I've just enabled gating tests for gfs2-utils and after the new build > finished I got a notification email from bodhi that said: > > "This update's test gating status has been changed to 'failed'." > > On the bodhi update page[0] the "fedora-ci.koji-build.tier0.functional" > test was highlighted red and marked as failed, with no extra information > available. > > After a while, the original 'failed' test turned green and showed > successful, which was puzzling. Why was it marked failed in the first > place? All's well that ends well, but I wasted a bit of time trying to > work out why it had failed when it really hadn't... Because the requirements aren't satisfied. The requirements are that certain specific tests pass. When Bodhi asks Greenwave what the status for a given update is, if there are passes for all of those tests, it says "passed". If not, it says "failed". That's all the subtlety there is. :) I suggested a while back that we should be able to define requirements in Greenwave as "if a result for this test exists, it must be pass" (but if no result exists, that's OK too). It hasn't been done, though, and some folks at least didn't think it was a good idea due to the possibility of allowing broken things through due to test names changing, issues in the test systems, etc. It also produces an obvious loophole that everything passes CI right after it's submitted, unless you start getting a bit hacky with minimum times to wait for results to appear and stuff, which starts to get complicated and possibly fragile. We could improve things somewhat here if we had an equivalent of ResultsDB (the source of truth for test results for this whole system) for test *execution*, but we don't. Any test system that wants into The System has to submit results to ResultsDB, but we don't have an equivalent test *execution* database that everything has to submit to. We do have a standard for text execution *messaging* - https://pagure.io/fedora-ci/messages - but not all test systems are entirely compliant with it, and using the info would require work in Greenwave and Bodhi and maybe elsewhere. I think Greenwave may provide some extended info here that Bodhi could use to be a bit more detailed (I *think* the 'extended' Greenwave result includes info on which tests are *failed* and which tests are *missing*); if it doesn't it could, and then Bodhi could actually draw a visual distinction between "there are failed tests" and "there are no failed tests but we don't have passes for all required tests yet". That might be an improvement. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
