On Sun, 2014-03-02 at 14:27 +0000, Richard W.M. Jones wrote: > On Fri, Feb 28, 2014 at 01:03:38PM -0800, Adam Williamson wrote: > > What I came up with is this gem: > > > > https://fedoraproject.org/wiki/User:Adamwill/Draft_storage_matrix > [...] > > Some of this is susceptible to automation, but some is not, in the sense > > that it involves the UI, and automated UI testing is a giant PITA. > > There was a *great* talk at either FOSDEM or DevConf about what > OpenSUSE is using for automation of their UIs. It's all open source, > easy to create the scripts using a GUI, and you can fiddle around with > bits of Javascript to fine tune the tests. > > Of course, now I cannot find the talk anywhere ... > > Anyone see / remember this? Don't really need it, we've already looked at SUSE's OpenQA thing. I saw it as soon as they announced it, and I've kept an eye on it since then. About once every two weeks *someone* mails me, another Fedora QA person, or test@ and says "hey, have you seen this OpenQA thing!??!?!" :P As GUI automated testing systems go, it's not a bad one. But it *does* suffer from the usual problems: mainly fragility. You can go to http://openqa.opensuse.org/results/ and note that the 'unknown' results are trending upwards over time. There are bad, mediocre and good GUI testing approaches out there, as with all things, but *even the good ones* need a lot of care and feeding. > > And 'susceptible to automation' is all very well, but it requires > > someone to do the work of automating it. > > While I'm not volunteering to implement this, wouldn't fuzz testing > using kickstarts be fairly easy to implement and automate? You > generate directed random kickstart files, run the installer using a > script similar to this: > > https://github.com/libguestfs/libguestfs/blob/master/builder/website/fedora.sh > > and then see whether the result is a plausible looking guest. (You > could use libguestfs to test whether the resultant guest contains > files at known locations, for example.) > > This moves the question of whether kickstart covers every possibility > in Anaconda (or vice versa, I suppose). But it has the advantage that > it seems like it would be easy to implement something quickly. It doesn't exercise the UI. Yes, you can use something like that to test the underlying partitioning code, and we really should be doing that and it's been on my todo list forever, but it still doesn't exercise the actual UI. Quite a lot of the bugs we hit tend to the classic UI edge case bugs, like "when you slide the slider all the way to the end it tries to create a partition that's 1 byte too small", that kind of thing. You can't really test those with a kickstart. You can, I believe, use a kickstart to produce absolutely any *result* you could get from the interactive installer, but you can't exercise all the UI codepaths. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
