On Sat, 2017-03-18 at 10:42 -0400, Colin Walters wrote: > On Fri, Mar 17, 2017, at 11:00 PM, Dusty Mabe wrote: > > > > Yes! we would love a UEFI test > > Yes; conceptually most of the important scenarios we run through > for Workstation/Server (e.g. dm-crypt versus not) we should also > do for Atomic Host. > > Adam, where is the git containing the input test suite for OpenQA? https://pagure.io/fedora-qa/os-autoinst-distri-fedora be aware, though, that the way tests work in openQA is somewhat idiosyncratic and very...specific. I can give you lots of details if you like, but the short version is that what we ultimately wind up actually *doing* so far as Atomic's concerned is: any time a fedmsg 'compose complete!' message appears for a compose that contains an Atomic Host installer image, openQA will spin up a VM with that ISO attached as an optical drive and a single empty hard disk, then do nearly the simplest possible install you can do. It just goes to INSTALLATION DESTINATION, selects the hard disk as the target and agrees to let anaconda partition it automatically, starts the install, sets a root password, and creates a user called 'test'. Once the install completes, it boots the installed system, logs in as root, checks whether there are any crash notifications or AVCs, uploads some resource usage information, and that's it. > > and I think it would actually be good > > to run the atomic-host-tests against these images assuming openQA is > > the right tool for that job. > > This part I don't quite agree with - remember the cloud images are > built with the installer ISO, and *those* are tested already with a-h-t. > There's going to be very little that's specific to the ISO path versus > the cloud image - which is part of the whole idea of sharing the > same exact ostree commit across them. > > > I thought openQA was more for interactive > > install testing, > > It does do more than that - however I see it as mostly valuable for testing > interactive Anaconda runs with the OpenCV approach. > > IMO OpenQA's use of OpenCV is less interesting for testing "headless/non-GUI" > bits, as is the case for Atomic Host. (We do of course have Cockpit > and the OpenShift web console, but both of those use Selenium[1][2] already) openQA is perfectly *capable* of doing non-graphical testing; we do actually do quite a lot of this just because it's there and relatively convenient to use. We don't use screen matching to do this, though. Quite a long time ago openQA grew the ability to do this kind of stuff through the serial console; you can use a convenience function to run any command at a console and either assert that it succeeds (which is implemented by echoing its return code plus a hash of the command - purely for identification purposes - to the serial console, then identifying when that text hits the serial console and extracting the return code), or do some kind of match on whatever the command sends to stdout. openQA doesn't have any huge benefits as a way of doing this compared to any of the other zillion implementations of 'spin up a VM and run some commands in it', though, beyond the fact that it can be useful to have a video recording of what was actually displayed on the console during the run (which is something very few other test systems provide). It has a couple of drawbacks, principally the idiosyncracies of how tests actually work in openQA; it's possible to hide this from people by writing some kinda middle layer code which would allow you to just dump a test script in a git repo and the middle layer code would automatically generate an openQA job that runs the script, I started writing something along those lines as a side project a while back just as a proof-of-concept that we could run tests from dist-git in openQA, but I've never finished it. If 'the atomic-host-tests' are just scripts that can be run from a console we could very trivially run them post-install as part of the openQA testing of the installer image, if desired. I agree with Colin that this is quite unlikely to produce different results from running the same tests on the equivalent Atomic host disk images in Taskotron or Autocloud, but it wouldn't really *hurt* anything (besides eating up a bit more of our limited openQA worker resources). We do also do some basic testing of Cockpit in openQA, though nowhere near as much as its own test suite covers. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ cloud mailing list -- cloud@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to cloud-leave@xxxxxxxxxxxxxxxxxxxxxxx
