On Mon, 2017-02-27 at 10:22 -0800, Adam Williamson wrote: > Hi folks! > > I am currently rolling out some changes to the Fedora openQA deployment > which enable a new testing workflow. From now on, a subset of openQA > tests should be run automatically on every critpath update, both on > initial submission and on any edit of the update. Hi again folks! Just a quick update on progress here so far. The deployment went pretty well, and the tests have been running now for the last week or so. You can view all the results so far here: https://openqa.fedoraproject.org/group_overview/2?limit_builds=400 One thing you might notice right away is the list sort order. openQA currently sorts 'builds' (in this context, the update is the 'build') on the assumption that they sort as dotted version strings, which Fedora update IDs (the string we use as the 'build' value for these tests) certainly don't. I've got a PR in progress upstream to allow us to sort these differently, and that should get changed soon. About half way through last week I implemented a change which means any failed test is automatically retried; this cut down quite a lot on false failures caused by transient bugs, mirror issues etc. There are still occasional cases of this, though. For now you can force all the tests to be re-run by editing the update in any way at all; in future we'll probably try and set up some system which lets you request re- runs of failed tests if the failures don't look like an actual bug in the update. This week I'm aiming to get the necessary changes made so that Bodhi will find and display these results alongside the Taskotron results in its web UI, which should make them much more visible. There's another significant factor which I hadn't considered: today was the Bodhi activation point for Fedora 26, meaning we now have Fedora 26 critpath updates we could test. For now I've decided to go ahead and try and test Branched updates, and just see how much of a mess it turns out to be. I suspect, though, that we'll have problems with the tests failing due to underlying bugs far more often (certainly several of the tests currently fail on Branched, for instance), and also we'll have problems with the base disk images much more often for pre-release branches. It may prove to be difficult or impossible to provide useful feedback for Branched updates with this system, and if so, we'll turn it off. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
