On Wed, 2019-02-06 at 16:44 -0500, Przemek Klosowski wrote: > On 2/6/19 4:17 PM, Adam Williamson wrote: > > Rawhide isn't really utterly broken: a GTK+ change broke lots of openQA > > needles (screenshots) so the tests are failing. I'm in the middle of > > re-taking all those now, and will re-run the tests on prod when it's > > done, and send a new compose check report. > > I am curious---if multiple tests fail this way, how do you determine > whether the changes are innocuous and re-taken screenshots can be used > from now on? > > I assume there is no automatic criterion, or else you'd be using it for > the test, so you just have to eyebal all of them, right? Essentially that, but openQA has mechanisms for making it easier. It has something called 'developer mode' which lets you run the test and, any time a screenshot match fails, pause and offer a 'needle editor' where you can go and create a new needle. You can base the new needle on the closest-matching existing needle, which copies all its attributes (including the match area) across, so all you have to do is eyeball and confirm the change is as expected, then hit save. Then you can resume the test from where you left off, and continue with that until all the needles are updated. Then you do the same for a bunch of other tests, till all the needles are done. :) Many needles are shared between tests, so it's not like you're redoing the same screens over and over - for each test you'll only have to redo any that haven't already been done. It took me about five hours to do all the new needles for this issue (62 of them), and that was with me doing other work in between as well. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
