On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> wrote:
We do sort of have this already, because openQA tests all updates, and
one of the tests it runs on them is an upgrade test.
Now, the iptables thing is interesting, because the logs of that test
on an F34 update show the issue:
2021-04-28T04:30:39-0400 WARNING
Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and iptables-libs-1.8.7-6.fc34.x86_64
- package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) = 1.8.7-3.fc34, but none of the providers can be installed
- cannot install the best update candidate for package iptables-libs-1.8.5-6.fc33.x86_64
- cannot install the best update candidate for package iptables-1.8.5-6.fc33.x86_64
but it's only logged as a WARNING and does not prevent the upgrade
running. So the test passes. It doesn't even need to use `--
allowerasing` so it doesn't register as a soft fail.
I assume that's because iptables from the main (fedora) repo are used instead. But the upgrade would fail if we added "--best" (I tested it). Here's a thought. What if we modified the openQA test to first try with "--best". If there's any error, log it as a softfail, and continue without it (and if if fails again, log it as a hard error). This way, we can catch *some* broken updates (it's still possible that one broken update obscures a second one and we'll not know about the latter - but the benefit is definitely there).
If we did it this way, what exactly happens with the softfailed result? Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing it soon enough?
> we could extend our upgrade testing matrices with two variants - one
would test with stable updates, the other would test with
updates-testing enabled
_______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
