Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> It sounds to me like the problem is "how do we best use the available
> automated test resources?" so I'll answer accordingly.  Ignore me if I
> misunderstood ;-)

No, not really, sorry if I didn't explain clearly enough :D It's more
just a 'what's the best way to work what I want to do around the
existing definitions' question. What I want to do is have updates of
FreeIPA-related packages gated. The awkwardness is that right now the
definition of what gets gated is "critical path packages", and those
clearly don't meet the current definition of "critical path".
> 
> We currently have a small list of packages that are gated behind openQA,
> and insufficient openQA resources to expand this list to all packages.
> 
> We also have a gating system based on karma, where users provide the QA
> manually.  At least, one hopes.  The karma system has some
> configurability for how much karma is needed, and for how long, etc.

Ah, no, again, I guess I should've explained that in more detail. When
I talk about "gating" I'm not talking about the manual karma mechanism.
I'm talking about the mechanism that prevents updates being pushed
stable if they fail the automated tests. That's a separate, parallel
thing which does not involve karma or manual feedback at all, it's
entirely automated.
> 
> What about a merger of those systems, where the openQA results count as
> a type of user, contributing to the status?  Give each package a "QA
> Priority" that ranges from "full gating" to "five second rule", where
> the openQA resources and the user work together to prioritize and test
> packages according to need:
> 
> * full gating requires all openQA tests to pass AND meet karma
>   requirements, openQA does these first.
> 
> * partial gating is like above, but either one (openQA or karma) can be
>   sufficient if enough time passes.
> 
> * reject only allows an openQA failure to reject an update, but the lack
>   of openQA success need not stop it if it has enough karma. (glibc uses
>   this paradigm - a ci/cd failure is an automatic reject, but a ci/cd
>   pass adds nothing to the consensus needed)
> 
> ... and so on down the list. Each user, including openQA, can "vote" a
> pass or fail, and the rules say how all those passes and fails result in
> the update's overall pass or fail.
> 
> This would allow the openQA resources to prioritize updates that it
> knows *needs* openQA results, which allocating spare cycles to packages
> that could benefit from testing but don't require it.  It also means
> that additional openQA resources can be put to use immediately, while
> still allowing for growth in critical path size, without being wasted on
> unseen results.

This is an interesting idea, but rather complex and doesn't solve any
problem I have right now. :P Sorry for the confusion!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux