On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote: > It sounds to me like the problem is "how do we best use the available > automated test resources?" so I'll answer accordingly. Ignore me if I > misunderstood ;-) No, not really, sorry if I didn't explain clearly enough :D It's more just a 'what's the best way to work what I want to do around the existing definitions' question. What I want to do is have updates of FreeIPA-related packages gated. The awkwardness is that right now the definition of what gets gated is "critical path packages", and those clearly don't meet the current definition of "critical path". > > We currently have a small list of packages that are gated behind openQA, > and insufficient openQA resources to expand this list to all packages. > > We also have a gating system based on karma, where users provide the QA > manually. At least, one hopes. The karma system has some > configurability for how much karma is needed, and for how long, etc. Ah, no, again, I guess I should've explained that in more detail. When I talk about "gating" I'm not talking about the manual karma mechanism. I'm talking about the mechanism that prevents updates being pushed stable if they fail the automated tests. That's a separate, parallel thing which does not involve karma or manual feedback at all, it's entirely automated. > > What about a merger of those systems, where the openQA results count as > a type of user, contributing to the status? Give each package a "QA > Priority" that ranges from "full gating" to "five second rule", where > the openQA resources and the user work together to prioritize and test > packages according to need: > > * full gating requires all openQA tests to pass AND meet karma > requirements, openQA does these first. > > * partial gating is like above, but either one (openQA or karma) can be > sufficient if enough time passes. > > * reject only allows an openQA failure to reject an update, but the lack > of openQA success need not stop it if it has enough karma. (glibc uses > this paradigm - a ci/cd failure is an automatic reject, but a ci/cd > pass adds nothing to the consensus needed) > > ... and so on down the list. Each user, including openQA, can "vote" a > pass or fail, and the rules say how all those passes and fails result in > the update's overall pass or fail. > > This would allow the openQA resources to prioritize updates that it > knows *needs* openQA results, which allocating spare cycles to packages > that could benefit from testing but don't require it. It also means > that additional openQA resources can be put to use immediately, while > still allowing for growth in critical path size, without being wasted on > unseen results. This is an interesting idea, but rather complex and doesn't solve any problem I have right now. :P Sorry for the confusion! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
