On Thu, 2011-10-13 at 08:35 -0400, Neal Becker wrote: > http://news.opensuse.org/2011/10/11/opensuse-announces-first-public-release-of- > openqa/ saw it! in fact, we've talked to one of the lead developers before. it's got some cool design features, a reasonably nice results interface, and it's somewhat more mature than AutoQA at present: these are good points about it. it relies on screenshot validation for pass/fail and it's written in freakin' perl: these are bad points about it. =) One problem with screenshot validation is that it's innately fragile: if we used such an approach for anaconda testing, for instance, we'd have to revise the 'pass' and 'fail' results for dozens of tests at least once a cycle, most likely. The entire suite would be invalidated by a UI rewrite like the one pending for F17. A consequence of the innate fragility is that it's very difficult to make it truly end-to-end automated; you can't just throw the raw results at developers because they're very often going to be invalid or incomplete. Whenever a test 'fails' or returns 'unknown' you need human intervention to take a look at it and verify what actually happened, then file an appropriate bug report. If you look at the bug reports down the left hand side of the openQA page, none of them was actually filed by an openQA bot, or anything: they're all hand-filed reports that have been done by people interpreting the openQA results. The AutoQA design is inherently more capable of allowing tests which can spit out results that are directly useful to developers without human intervention; in fact, depcheck already does so, by giving the actual yum output describing the dependency issues it finds. The other problem with screenshot validation is that it's inherently unsuited to some tests that are important to Fedora and that we have more-or-less working *right now* in AutoQA - things like depcheck and repoclosure - because all it can really tell you is 'does this results screen look like a known pass case' - it's inherently pass/fail based, which is no use if the output you want is "these five dependencies are broken". But hey, it's dumb but it works right now, which means it's not dumb: in a sense, updating a screenshot hash once a week is still much better than manually re-doing the test every day, and though the approach doesn't work for _some_ tests, there are certainly tests that would be useful to Fedora which we _could_ implement with screenshot validation. The test format of openQA is nice and idiot-proof too, at least from the quick look I took at it - each test can have as many known 'pass' and 'fail' screenshot hashes as you want, and adding one is literally a one-line copy/paste/modify operation, so it would be very contributable-to. We're about to hit the F16 final crunch, at which point QA has about zero time to look at anything but release validation, but once final is done we could certainly try and find someone to take a quick look at openQA and see if it would be viable to make it usable for at least a bit of Fedora testing in the short term. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
