On Fri, 2015-10-09 at 21:41 +0200, Kalev Lember wrote: > These failed test IDs are all nice to see, but they don't say much to > the uninitiated. :) Yeah, what I'd actually like to do is have links to the tests; the reason we can't do that right now is that the openQA instance is behind the RH firewall so only RHers can see it. We're trying to fix that - in fact that's what I'd be doing today if I didn't have to retake a few dozen screenshots instead :P > These tests are comparing screenshots pixel by > pixel, right? Are the screenshots and other test results publicly > available somewhere? See above. And soooooorrt of, but not really. openQA matches on areas within screenshots; an openQA 'needle' is a screenshot with an accompany JSON file that identifies areas within it (and contains tags used for actually doing matches). A needle 'matches' if all the marked areas have matches within the screenshot being compared, after a bit of image processing is applied (color reduction and sharpening), within a degree of tolerance (~96% similarity according to some algorithm or other). > How are the screenshots rebased? Is it all manual work? More or less. the openQA web UI has an interactive needle editor which lets you basically run a test and pause after each failed match, then you can fiddle with the needle and re-run the match. I haven't used it much and never for doing a big screenshot rebase, so I can't say whether it's more or less convenient than the way I did the last one, which was to reproduce the tests by hand in a VM, take the screenshots from virt-manager, copy the json file and make any necessary adjustments to the match area with the 'crop.py' needle editor (which is an alternative tool upstream provides in one of their repos). > Is there any way to link to the failed screenshots from the status > mail > so that I could just follow links and see what's changed? Once we get openQA deployed publicly, yeah. As I said, we're working on that; https://phab.qadevel.cloud.fedoraproject.org/T623 for the gritty details. > Sorry for bombing you with all the questions. I think it's totally > awesome that we have nightly testing, thanks for getting this set up! No probs! Ultimately this is an occupational hazard of how openQA works, it's just that I was surprised that a font would suddenly change its appearance quite noticeably (at least to a tool like openQA). But now I got some references from you and Matthias I understand that it was to fix clear rendering errors, which is pretty reasonable, I just hope there isn't too much churn in future releases :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
