Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> It sounds to me like the problem is "how do we best use the available
> automated test resources?" so I'll answer accordingly.  Ignore me if I
> misunderstood ;-)

No, not really, sorry if I didn't explain clearly enough :D It's more
just a 'what's the best way to work what I want to do around the
existing definitions' question. What I want to do is have updates of
FreeIPA-related packages gated. The awkwardness is that right now the
definition of what gets gated is "critical path packages", and those
clearly don't meet the current definition of "critical path".
> We currently have a small list of packages that are gated behind openQA,
> and insufficient openQA resources to expand this list to all packages.
> We also have a gating system based on karma, where users provide the QA
> manually.  At least, one hopes.  The karma system has some
> configurability for how much karma is needed, and for how long, etc.

Ah, no, again, I guess I should've explained that in more detail. When
I talk about "gating" I'm not talking about the manual karma mechanism.
I'm talking about the mechanism that prevents updates being pushed
stable if they fail the automated tests. That's a separate, parallel
thing which does not involve karma or manual feedback at all, it's
entirely automated.
> What about a merger of those systems, where the openQA results count as
> a type of user, contributing to the status?  Give each package a "QA
> Priority" that ranges from "full gating" to "five second rule", where
> the openQA resources and the user work together to prioritize and test
> packages according to need:
> * full gating requires all openQA tests to pass AND meet karma
>   requirements, openQA does these first.
> * partial gating is like above, but either one (openQA or karma) can be
>   sufficient if enough time passes.
> * reject only allows an openQA failure to reject an update, but the lack
>   of openQA success need not stop it if it has enough karma. (glibc uses
>   this paradigm - a ci/cd failure is an automatic reject, but a ci/cd
>   pass adds nothing to the consensus needed)
> ... and so on down the list. Each user, including openQA, can "vote" a
> pass or fail, and the rules say how all those passes and fails result in
> the update's overall pass or fail.
> This would allow the openQA resources to prioritize updates that it
> knows *needs* openQA results, which allocating spare cycles to packages
> that could benefit from testing but don't require it.  It also means
> that additional openQA resources can be put to use immediately, while
> still allowing for growth in critical path size, without being wasted on
> unseen results.

This is an interesting idea, but rather complex and doesn't solve any
problem I have right now. :P Sorry for the confusion!
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha

test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct:
List Guidelines:
List Archives:
Do not reply to spam, report it:

[Index of Archives]     [Fedora Desktop]     [Fedora SELinux]     [Photo Sharing]     [Yosemite Forum]     [KDE Users]

  Powered by Linux