On Tue, 2022-08-30 at 09:39 +0300, Alexander Bokovoy wrote: > On ma, 29 elo 2022, Adam Williamson wrote: > > On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote: > > > It sounds to me like the problem is "how do we best use the available > > > automated test resources?" so I'll answer accordingly. Ignore me if I > > > misunderstood ;-) > > > > No, not really, sorry if I didn't explain clearly enough :D It's more > > just a 'what's the best way to work what I want to do around the > > existing definitions' question. What I want to do is have updates of > > FreeIPA-related packages gated. The awkwardness is that right now the > > definition of what gets gated is "critical path packages", and those > > clearly don't meet the current definition of "critical path". > > Not all of them meet the current definition of 'critical path' but many > do. In fact, almost all crypto-implementing packages should be on that > list, if not already. Sure, but freeipa itself, for instance, can't really be as things stand. Nor could bind, I don't think. And several others. > > Ideally, use of gating tests should be defined in the component itself. > Do we have a mechanism to trigger OpenQA runs from the gating.yaml in > the specific component? If we do, then we really should move it there. No. I don't think this is in-scope for gating.yaml. I *could* extend the openQA scheduler to check some kind of per-repo config for every package in the update, but there is nothing like this at present. In any case, triggering the *tests* isn't the problem here; we already do that, with the allowlist. It's doing the *gating* that's the problem. As I said, we could add *that* to gating.yaml for each repo, but I don't really love that idea, because you have to provide a full list of what tests to gate on, and doing that for each repo and keeping it updated seems like kind of a drag. I guess in a sense it's appropriate for these packages, though, as you could say we'd "really" want to gate on the FreeIPA-specific tests, or at least just the server tests. But honestly, if an update to a freeipa-related package causes GNOME to start crashing or something, that's something I want to look into before the update goes stable, even if it seems very weird. > > Another part of the problem is that most of these runs should probably > be consulting, not gating in the sense that failing them should give you > a clue that things might be wrong but there needs to be a human overview > of the failures. It is what we have right now with OpenQA-reported > failures for Bodhi updates in most cases, the test results aren't really > preventing the submissions from the going forward unless a maintainer > defined the update configuration to be so. The problem cases here are what I was thinking about. They are: Branched before Beta freeze, and Rawhide. For those cases, updates are automatically created on `fedpkg build` and immediately submitted for stable. If there is no applicable gating policy they are immediately pushed stable. There is no time for anyone to consult. If you use a side tag there's no automatic submission, I don't think, but people do not use side tags as a matter of course. Right now we don't gate Rawhide updates, but I would like to and I am actively working on it (this would prevent me having the kind of nightmare week I had last week where I spent dozens of hours tracking down things that had broken in Rawhide, getting them fixed, and re- running tests). We *do* currently gate Branched updates before Beta freeze, and I want FreeIPA-related packages included in that for the next cycle. I would have liked it if, for the F37 cycle, FreeIPA-related updates that were submitted between Branch point and Beta freeze were gated. But they weren't. Practically speaking, the way we do things in Fedora, there is very little difference between "gating" and "consulting"; because of the waiver mechanism they're effectively identical. You can always issue waivers to override the gating. So really, gating is consulting. You can look at the failures, decide they're bogus, and hit the waiver button, which will immediately make the update 'pass' gating. I do encourage people not to abuse the waiver function - if the failure seems like a flake please use the 're-run tests' button, if you really think it's bogus please poke me or Lukas or someone as I'm probably working right then on dealing with it - but it is right there. > > Ideally, we should have reverse dependency updates to trigger FreeIPA > tests and give their results a visibility but don't block on failures. > This would give an opportunnity to notice a failurer earlier but don't > discourage maintainers from participating in a joint development. We would like reverse dependency testing for all sorts of reasons, but it's also all sorts of difficult (and, potentially, resource- intensive). Of course, in openQA testing we do sort of have this already. We run *all* tests on all critpath updates. So if an update to a critpath component breaks FreeIPA, we find out about it immediately. We don't have to wait for a FreeIPA-related update to show up. If the dep isn't critpath, though, we don't test it. > Perhaps, The Fedora Packager Dashboard could be extended to pick up > results of tests relevant to your packages and display them together? > This way FreeIPA maintainers can see an overview of all tests related to > FreeIPA in one place and see if any help is needed to resolve failures. This would be lovely, sure, but at this point we're quite far away from the fairly concrete initial scope of my mail. I don't want to get sidetracked into sky castle building where we file tickets and wait two years for people to build stuff. I would like to gate updates to FreeIPA-related packages *now*. I was all set to submit a pull request to comps to add them to the server critpath definition, but then I checked the official definition of "critical path" and noticed this problem. That's where I'm coming from here: I want to make a concrete change now, not design awesome speculative improvements on paper. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
