On Wed, 2019-02-06 at 13:59 -0800, Adam Williamson wrote: > On Wed, 2019-02-06 at 16:44 -0500, Przemek Klosowski wrote: > > On 2/6/19 4:17 PM, Adam Williamson wrote: > > > Rawhide isn't really utterly broken: a GTK+ change broke lots of openQA > > > needles (screenshots) so the tests are failing. I'm in the middle of > > > re-taking all those now, and will re-run the tests on prod when it's > > > done, and send a new compose check report. > > > > I am curious---if multiple tests fail this way, how do you determine > > whether the changes are innocuous and re-taken screenshots can be used > > from now on? > > > > I assume there is no automatic criterion, or else you'd be using it for > > the test, so you just have to eyebal all of them, right? > > Essentially that, but openQA has mechanisms for making it easier. It > has something called 'developer mode' which lets you run the test and, > any time a screenshot match fails, pause and offer a 'needle editor' > where you can go and create a new needle. You can base the new needle > on the closest-matching existing needle, which copies all its > attributes (including the match area) across, so all you have to do is > eyeball and confirm the change is as expected, then hit save. Then you > can resume the test from where you left off, and continue with that > until all the needles are updated. Then you do the same for a bunch of > other tests, till all the needles are done. :) > > Many needles are shared between tests, so it's not like you're redoing > the same screens over and over - for each test you'll only have to redo > any that haven't already been done. > > It took me about five hours to do all the new needles for this issue > (62 of them), and that was with me doing other work in between as well. Oh, also note/remember that openQA doesn't really match entire screenshots, it matches *specified areas* of them. And the interfaces mentioned above have good tools for highlighting the match areas and showing the difference between what was expected and what was seen. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
