Re: New automated test coverage: openQA tests of critical path updates

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, 2017-02-27 at 10:22 -0800, Adam Williamson wrote:
> Hi folks!
> 
> I am currently rolling out some changes to the Fedora openQA deployment
> which enable a new testing workflow. From now on, a subset of openQA
> tests should be run automatically on every critpath update, both on
> initial submission and on any edit of the update.

Hi again folks! Just a quick update on progress here so far.

The deployment went pretty well, and the tests have been running now
for the last week or so. You can view all the results so far here:

https://openqa.fedoraproject.org/group_overview/2?limit_builds=400

One thing you might notice right away is the list sort order. openQA
currently sorts 'builds' (in this context, the update is the 'build')
on the assumption that they sort as dotted version strings, which
Fedora update IDs (the string we use as the 'build' value for these
tests) certainly don't. I've got a PR in progress upstream to allow us
to sort these differently, and that should get changed soon.

About half way through last week I implemented a change which means any
failed test is automatically retried; this cut down quite a lot on
false failures caused by transient bugs, mirror issues etc. There are
still occasional cases of this, though. For now you can force all the
tests to be re-run by editing the update in any way at all; in future
we'll probably try and set up some system which lets you request re-
runs of failed tests if the failures don't look like an actual bug in
the update.

This week I'm aiming to get the necessary changes made so that Bodhi
will find and display these results alongside the Taskotron results in
its web UI, which should make them much more visible.

There's another significant factor which I hadn't considered: today was
the Bodhi activation point for Fedora 26, meaning we now have Fedora 26
critpath updates we could test.

For now I've decided to go ahead and try and test Branched updates, and
just see how much of a mess it turns out to be. I suspect, though, that
we'll have problems with the tests failing due to underlying bugs far
more often (certainly several of the tests currently fail on Branched,
for instance), and also we'll have problems with the base disk images
much more often for pre-release branches. It may prove to be difficult
or impossible to provide useful feedback for Branched updates with this
system, and if so, we'll turn it off.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
_______________________________________________
test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx




[Index of Archives]     [Fedora Desktop]     [Fedora SELinux]     [Photo Sharing]     [Yosemite Forum]     [KDE Users]

  Powered by Linux