Re: Plan / proposal: enable openQA update testing and potentially gating on Rawhide updates

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, 2023-06-12 at 12:05 +0200, Petr Pisar wrote:
> V Fri, Jun 09, 2023 at 11:38:51AM -0700, Adam Williamson napsal(a):
> > On Thu, 2022-06-09 at 12:48 -0700, Adam Williamson wrote:
> > > Hi folks!
> > > 
> > > More significantly, I'd also propose that we turn on gating on openQA
> > > results for Rawhide updates. This would mean Rawhide updates would be
> > > held from going 'stable' (and included in the next compose) until the
> > > gating openQA tests had run and passed. We may want to do this a bit
> > > after turning on the tests; perhaps Fedora 37 branch point would be a
> > > natural time to do it.
> > 
> > Hi again folks! A quick update here. Now the Rawhide update testing has
> > been running in production for over a year - and Kevin and I have been
> > "shadow gating" Rawhide for several months, untagging updates where
> > openQA tests indicate genuine bugs - I think it's time to go ahead and
> > enable gating for Rawhide updates. I've worked to make sure the tests
> > are reliable and failures are promptly investigated, and that Bodhi
> > provides accurate information on test and gating status. I've proposed
> > this as a FESCo ticket just to get some visibility and sign-off on the
> > idea:
> > 
> OpenQA update testing are the graphical tests, e.g. update.live_build test
> <https://openqa.fedoraproject.org/tests/1939328>?
> 
> I always strugle to understand the results. What does a gray frame mean? In
> failed (red?) frame, how can I see the expected image? The frame is usually
> split by a vertical line, so I guess one side is expected, other side is
> observed. Which one is one? What does mean "Test died: no candidate needle
> with tag(s) '...' matched" message?

I can give you detailed answers for all of these, but the real answer
is: I or someone else from the QA team will investigate failures and
process them into more useful information. We don't expect packagers to
diagnose their own failures from the openQA results unless they're very
keen. We see openQA testing as a service provided by the QA team, and
part of the service is investigating and diagnosing (and, quite often,
fixing) failures. It's not intended a self-service system like Fedora
CI.

One thing you should look for in the UI is the Comments tab: usually
once we've looked into a failure we will drop a link there to explain
it (typically a link to a bug report or pull request or issue).

Short answers:

* Grey frame usually just means "didn't find the expected image, but
timer hasn't expired yet" - expected events in openQA have a timeout,
and it logs attempts to match during that period for information.

* Vertical line - yes, I think it's left side == old, right side ==
new, but note the diff shown is *only* the attempted match area(s)
(which are outlined) by default; clicking the 'eye' icon changes this
to a full image diff.

* Message just means "openQA was looking for a screenshot match and
didn't find it" - in openQA parlance, the expected screenshot plus
metadata (what areas to match etc.) is called a "needle", and part of
the metadata is the "tag(s)". This is just a standard flexibility
thing, instead of being forced to match a single needle by its single
name, you match a "tag", and multiple needles can provide the same tag
and any single needle can provide multiple different tags, which gives
several obvious benefits.

> And most importantly, who should I contact to help me investigate the
> failures.

The QA team, however you want to do that - #fedora-qa on IRC / Matrix,
test@ mailing list, #quality-team tag on Discourse, poke me on
Mastodon, email, carrier pigeon...usually you can also poke me
directly, if I'm not around then Lukas Ruzicka (lruzicka). I usually
check openQA update test results four or five times a day on weekdays
(west coast NA time) and at least once a day on weekends.

>   As a packager, I need to retract the update or waive it promptly.
> That means either the packager needs to be able to reproduce the failure
> locally (how?)

This is possible, but it's not gonna be practical for most packagers.
Standing up an openQA instance requires a decent amount of work and
resources; you can use the official ansible plays -
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openqa plus
supporting stuff in the group_vars and so on - to help, but even then
it's still not trivial. FWIW, I don't even have a pet openQA instance
any more (Lukas does); I just use the official staging instance for re-
runs and fiddling around with stuff...

>  or there must be some one else to do it.

We (QA team) will do this.

> I would expect answers for these questions to be available in a documentation.
> Yet, I can any pointer to the documentation from a page with the results.

That's a good idea, I'll see if we can work in a link somewhere in
Bodhi, the openQA UI, or both.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@xxxxxxxxxxxxx
https://www.happyassassin.net



_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux