On ti, 30 elo 2022, Adam Williamson wrote:
On Tue, 2022-08-30 at 09:39 +0300, Alexander Bokovoy wrote:
On ma, 29 elo 2022, Adam Williamson wrote:
> On Tue, 2022-08-30 at 00:32 -0400, DJ Delorie wrote:
> > It sounds to me like the problem is "how do we best use the available
> > automated test resources?" so I'll answer accordingly. Ignore me if I
> > misunderstood ;-)
>
> No, not really, sorry if I didn't explain clearly enough :D It's more
> just a 'what's the best way to work what I want to do around the
> existing definitions' question. What I want to do is have updates of
> FreeIPA-related packages gated. The awkwardness is that right now the
> definition of what gets gated is "critical path packages", and those
> clearly don't meet the current definition of "critical path".
Not all of them meet the current definition of 'critical path' but many
do. In fact, almost all crypto-implementing packages should be on that
list, if not already.
Sure, but freeipa itself, for instance, can't really be as things
stand. Nor could bind, I don't think. And several others.
Ideally, use of gating tests should be defined in the component itself.
Do we have a mechanism to trigger OpenQA runs from the gating.yaml in
the specific component? If we do, then we really should move it there.
No. I don't think this is in-scope for gating.yaml. I *could* extend
the openQA scheduler to check some kind of per-repo config for every
package in the update, but there is nothing like this at present.
In any case, triggering the *tests* isn't the problem here; we already
do that, with the allowlist. It's doing the *gating* that's the
problem. As I said, we could add *that* to gating.yaml for each repo,
but I don't really love that idea, because you have to provide a full
list of what tests to gate on, and doing that for each repo and keeping
it updated seems like kind of a drag.
I guess in a sense it's appropriate for these packages, though, as you
could say we'd "really" want to gate on the FreeIPA-specific tests, or
at least just the server tests. But honestly, if an update to a
freeipa-related package causes GNOME to start crashing or something,
that's something I want to look into before the update goes stable,
even if it seems very weird.
From FreeIPA side I'd split this into two parts:
- turn allowedlist into critpath list and treat all these tests as
gating ones properly, run them for all cases
- finally make FreeIPA CI tests to run on Fedora updates and make them
gating too.
There are hundreds of tests we run in RHEL and in FreeIPA CI upstream
that Fedora would benefit from because they excercise more unlit corners
than OpenQA does.
I agree with your arguments below, we need to turn gating for all use
cases, especially Branched before Beta freeze, and Rawhide. Resource
wise it may be excessive but early detection of complex issues FreeIPA
testing tends to uncover is worth it.
Practically speaking, the way we do things in Fedora, there is very
little difference between "gating" and "consulting"; because of the
waiver mechanism they're effectively identical. You can always issue
waivers to override the gating. So really, gating is consulting. You
can look at the failures, decide they're bogus, and hit the waiver
button, which will immediately make the update 'pass' gating. I do
encourage people not to abuse the waiver function - if the failure
seems like a flake please use the 're-run tests' button, if you really
think it's bogus please poke me or Lukas or someone as I'm probably
working right then on dealing with it - but it is right there.
Do you have measured data for waivers? How often the waiver process is
used at all? In Branched before beta freeze?
This would be lovely, sure, but at this point we're quite far away from
the fairly concrete initial scope of my mail. I don't want to get
sidetracked into sky castle building where we file tickets and wait two
years for people to build stuff. I would like to gate updates to
FreeIPA-related packages *now*. I was all set to submit a pull request
to comps to add them to the server critpath definition, but then I
checked the official definition of "critical path" and noticed this
problem. That's where I'm coming from here: I want to make a concrete
change now, not design awesome speculative improvements on paper.
So your real issue at the moment is that 'critical path' definition is
not helping to QE. I guess, it is a worth submission to FESCO then,
right?
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
