Thoughts welcome: interface between automated test gating and the "critical path"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi folks!

I have one of those definitional quandaries and I figured I'd throw it
at the lists for some input.

Right now, we kind of take advantage of the "critical path" concept for
automated update testing and gating via openQA.

openQA does not test all updates, only critical path updates plus
updates containing any package on a short allowlist.

Bodhi and Greenwave do not gate all updates on the openQA tests since
not all updates are tested; again we use the critpath definition. If an
update is critical path, it gets gated on the openQA tests. If it
isn't, it doesn't.

If you've been paying attention, that means there's a bit of a hole:
the packages on the 'allowlist'. These are tested, but not gated.

What's on the allowlist? Basically, FreeIPA-related packages. We have a
good set of FreeIPA tests in openQA, and both I and the FreeIPA team
wanted relevant updates to be tested with them. But those packages are
not on the critical path. So I set up this 'allowlist' mechanism.

However, the lack of gating kinda sucks. I would much prefer if we
could gate these updates as well as just testing them.

The obvious thing to do is add the FreeIPA-related packages to the
critical path, but that really is not supported by the critical path
definition:

https://fedoraproject.org/wiki/Critical_path_package

which defines it as packages required to "perform the most fundamental
actions on a system", with a list of specific areas, none of which is
"run a domain server".

So...what to do?

I can think of I guess four options:

1. Broaden the definition of the "critical path" somehow. We could just
write in that it includes FreeIPA functionality, I guess, though that
seems special purpose. We could broaden it to include any functionality
covered by the release criteria, which would be quite a big increase
but seems like a reasonable and concise definition. Or we could write
in that it includes any functionality that is exercised by the gating
openQA tests, though that seems a bit arbitrary - there's no
particularly great organizing principle to the openQA tests we choose
to run on updates, if I'm honest, it's a sort of grab bag I came up
with.

2. Keep the current "critical path" concept but define a broader group
of "gated packages" somewhere (could be comps or somewhere else). This
would require engineering work - we'd have to touch probably the openQA
scheduler, Bodhi, and greenwave configs. It's also another maintenance
burden.

3. Add gating config to each allowlisted package repo's gating.yml one
by one. I don't really like this option. It'd be a chunk of work to do
initially, and multiples the maintenance required when the list of
tests to gate on changes for some reason.

4. Do nothing, just keep only gating things that are "critical path" on
the current definition. In a utopian future, we'd manage to deploy
openQA in the cloud or get a giant pile of super fast servers so we
could test and gate *every* update, and this wouldn't be an issue any
more.

What do folks think? Any preferences? Thanks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

_______________________________________________
test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Index of Archives]     [Fedora Desktop]     [Fedora SELinux]     [Photo Sharing]     [Yosemite Forum]     [KDE Users]

  Powered by Linux