Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



It sounds to me like the problem is "how do we best use the available
automated test resources?" so I'll answer accordingly.  Ignore me if I
misunderstood ;-)

We currently have a small list of packages that are gated behind openQA,
and insufficient openQA resources to expand this list to all packages.

We also have a gating system based on karma, where users provide the QA
manually.  At least, one hopes.  The karma system has some
configurability for how much karma is needed, and for how long, etc.

What about a merger of those systems, where the openQA results count as
a type of user, contributing to the status?  Give each package a "QA
Priority" that ranges from "full gating" to "five second rule", where
the openQA resources and the user work together to prioritize and test
packages according to need:

* full gating requires all openQA tests to pass AND meet karma
  requirements, openQA does these first.

* partial gating is like above, but either one (openQA or karma) can be
  sufficient if enough time passes.

* reject only allows an openQA failure to reject an update, but the lack
  of openQA success need not stop it if it has enough karma. (glibc uses
  this paradigm - a ci/cd failure is an automatic reject, but a ci/cd
  pass adds nothing to the consensus needed)

... and so on down the list. Each user, including openQA, can "vote" a
pass or fail, and the rules say how all those passes and fails result in
the update's overall pass or fail.

This would allow the openQA resources to prioritize updates that it
knows *needs* openQA results, which allocating spare cycles to packages
that could benefit from testing but don't require it.  It also means
that additional openQA resources can be put to use immediately, while
still allowing for growth in critical path size, without being wasted on
unseen results.
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux