Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, 2022-08-30 at 09:39 +0300, Alexander Bokovoy wrote:
> On ma, 29 elo 2022, Adam Williamson wrote:
> > On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> > > It sounds to me like the problem is "how do we best use the available
> > > automated test resources?" so I'll answer accordingly.  Ignore me if I
> > > misunderstood ;-)
> > 
> > No, not really, sorry if I didn't explain clearly enough :D It's more
> > just a 'what's the best way to work what I want to do around the
> > existing definitions' question. What I want to do is have updates of
> > FreeIPA-related packages gated. The awkwardness is that right now the
> > definition of what gets gated is "critical path packages", and those
> > clearly don't meet the current definition of "critical path".
> 
> Not all of them meet the current definition of 'critical path' but many
> do. In fact, almost all crypto-implementing packages should be on that
> list, if not already.

Sure, but freeipa itself, for instance, can't really be as things
stand. Nor could bind, I don't think. And several others.
> 
> Ideally, use of gating tests should be defined in the component itself.
> Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
> the specific component? If we do, then we really should move it there.

No. I don't think this is in-scope for gating.yaml. I *could* extend
the openQA scheduler to check some kind of per-repo config for every
package in the update, but there is nothing like this at present.

In any case, triggering the *tests* isn't the problem here; we already
do that, with the allowlist. It's doing the *gating* that's the
problem. As I said, we could add *that* to gating.yaml for each repo,
but I don't really love that idea, because you have to provide a full
list of what tests to gate on, and doing that for each repo and keeping
it updated seems like kind of a drag.

I guess in a sense it's appropriate for these packages, though, as you
could say we'd "really" want to gate on the FreeIPA-specific tests, or
at least just the server tests. But honestly, if an update to a
freeipa-related package causes GNOME to start crashing or something,
that's something I want to look into before the update goes stable,
even if it seems very weird.
> 
> Another part of the problem is that most of these runs should probably
> be consulting, not gating in the sense that failing them should give you
> a clue that things might be wrong but there needs to be a human overview
> of the failures. It is what we have right now with OpenQA-reported
> failures for Bodhi updates in most cases, the test results aren't really
> preventing the submissions from the going forward unless a maintainer
> defined the update configuration to be so.

The problem cases here are what I was thinking about. They are:
Branched before Beta freeze, and Rawhide.

For those cases, updates are automatically created on `fedpkg build`
and immediately submitted for stable. If there is no applicable gating
policy they are immediately pushed stable. There is no time for anyone
to consult.

If you use a side tag there's no automatic submission, I don't think,
but people do not use side tags as a matter of course.

Right now we don't gate Rawhide updates, but I would like to and I am
actively working on it (this would prevent me having the kind of
nightmare week I had last week where I spent dozens of hours tracking
down things that had broken in Rawhide, getting them fixed, and re-
running tests).

We *do* currently gate Branched updates before Beta freeze, and I want
FreeIPA-related packages included in that for the next cycle. I would
have liked it if, for the F37 cycle, FreeIPA-related updates that were
submitted between Branch point and Beta freeze were gated. But they
weren't.

Practically speaking, the way we do things in Fedora, there is very
little difference between "gating" and "consulting"; because of the
waiver mechanism they're effectively identical. You can always issue
waivers to override the gating. So really, gating is consulting. You
can look at the failures, decide they're bogus, and hit the waiver
button, which will immediately make the update 'pass' gating. I do
encourage people not to abuse the waiver function - if the failure
seems like a flake please use the 're-run tests' button, if you really
think it's bogus please poke me or Lukas or someone as I'm probably
working right then on dealing with it - but it is right there.
> 
> Ideally, we should have reverse dependency updates to trigger FreeIPA
> tests and give their results a visibility but don't block on failures.
> This would give an opportunnity to notice a failurer earlier but don't
> discourage maintainers from participating in a joint development.

We would like reverse dependency testing for all sorts of reasons, but
it's also all sorts of difficult (and, potentially, resource-
intensive). Of course, in openQA testing we do sort of have this
already. We run *all* tests on all critpath updates. So if an update to
a critpath component breaks FreeIPA, we find out about it immediately.
We don't have to wait for a FreeIPA-related update to show up. If the
dep isn't critpath, though, we don't test it.

> Perhaps, The Fedora Packager Dashboard could be extended to pick up
> results of tests relevant to your packages and display them together?
> This way FreeIPA maintainers can see an overview of all tests related to
> FreeIPA in one place and see if any help is needed to resolve failures.

This would be lovely, sure, but at this point we're quite far away from
the fairly concrete initial scope of my mail. I don't want to get
sidetracked into sky castle building where we file tickets and wait two
years for people to build stuff. I would like to gate updates to
FreeIPA-related packages *now*. I was all set to submit a pull request
to comps to add them to the server critpath definition, but then I
checked the official definition of "critical path" and noticed this
problem. That's where I'm coming from here: I want to make a concrete
change now, not design awesome speculative improvements on paper.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux