On Thu, 2021-04-29 at 10:09 +0200, Kamil Paral wrote: > On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson <adamwill@xxxxxxxxxxxxxxxxx> > wrote: > > > We do sort of have this already, because openQA tests all updates, and > > one of the tests it runs on them is an upgrade test. > > > > Now, the iptables thing is interesting, because the logs of that test > > on an F34 update show the issue: > > > > 2021-04-28T04:30:39-0400 WARNING > > Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and > > iptables-libs-1.8.7-6.fc34.x86_64 > > - package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) = > > 1.8.7-3.fc34, but none of the providers can be installed > > - cannot install the best update candidate for package > > iptables-libs-1.8.5-6.fc33.x86_64 > > - cannot install the best update candidate for package > > iptables-1.8.5-6.fc33.x86_64 > > > > but it's only logged as a WARNING and does not prevent the upgrade > > running. So the test passes. It doesn't even need to use `-- > > allowerasing` so it doesn't register as a soft fail. > > > > I assume that's because iptables from the main (fedora) repo are used > instead. But the upgrade would fail if we added "--best" (I tested it). > Here's a thought. What if we modified the openQA test to first try with > "--best". If there's any error, log it as a softfail, and continue without > it (and if if fails again, log it as a hard error). This way, we can catch > *some* broken updates (it's still possible that one broken update obscures > a second one and we'll not know about the latter - but the benefit is > definitely there). > > If we did it this way, what exactly happens with the softfailed result? > Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing > it soon enough? So I looked into this some more yesterday. openQA did in fact catch this bug, straight up: when the iptables update was initially submitted, an openQA upgrade test failed on exactly this problem. The upgrade succeeded, but the test failed because of the last-step check we do in openQA that, if any package from the update is installed, it is the version from the update that is installed. Because the upgrade pulled in the earlier -3 version and not the -6 version from the update, the test failed. I mentioned this in Bodhi, even, but I noted that if the update was pushed stable *before* F34's final release, the bug should disappear, because the -3 builds would no longer be available as options and dnf would in that case choose to go with -6 and remove iptables (I tested this). Unfortunately, the update was *not* pushed stable before final release, so the -3 builds are in the frozen 'fedora' repo forever now, and we have a more or less insoluble problem (probably the best option is just to put the iptables package back in F34, and re-do this better for F35). See the bug report for more details: https://bugzilla.redhat.com/show_bug.cgi?id=1953178 Given this, I don't think we need to use your idea, since it's demonstrated that openQA *does* actually catch bugs like this already. We just need maintainers to pay more attention and follow up better. > > As for manual testing, what do you think about my previous proposal? > > we could extend our upgrade testing matrices with two variants - one > would test with stable updates, the other would test with updates-testing > enabled Sure, it's possible. It just adds more testing that either a human or a robot needs to do. And, again, it turns out our current testing did actually catch this problem just fine. If anything went wrong, it was the follow-up. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
