Hi folks! Just wanted to drop a note for anyone who noticed their Rawhide updates stuck in gating for an unusually long time yesterday/today. This is mainly due to the mass rebuild. Merging it in (which was done without sending anything in it through openQA testing) resulted in two separate problems: 1. Some GNOME packages were inadvertently bumped to 45-alpha early, as the changes were checked into git but had not been built; this caused dependency issues, then when we tried building the remaining packages at version 45-alpha to see if things would work OK with a consistent set of packages, we found a bug which made the tests fail and necessitated untagging everything back to 44: https://gitlab.gnome.org/GNOME/mutter/-/issues/2918 . 2. The rebuilt PackageKit was broken because of a change in glib, and needed a backport of https://github.com/PackageKit/PackageKit/pull/643 to work properly again. These were complicated by a third problem showing up in the middle: a new build of brltty had some internal dependency issues, and needed to be untagged. All of these issues caused test failures for *every* Rawhide update tested after the mass rebuild was merged (in the first case) or the brltty update landed (in the third case). It took several hours to get them all unpicked and the fixed PackageKit landed, then I re-scheduled the failed tests on all other updates (but I did this a bit wrong for some of them, so they failed again, and I had to re-run them again an hour or so ago). I'm also fighting a bit with Koji giving 404 responses for the f39- build repo more often than it usually does, but I hope to have tests for all updates passing (assuming the update itself has no problems) within the next hour or two. For the next go-round, it might be good to use the relatively new ability we have to schedule openQA tests on a side tag to test the mass rebuild *before* it's merged, I'll check with releng if that is feasible. This is also the second or third time gating has been disrupted by a broken update that wasn't in the critical path and so didn't go through gating itself. I'm considering rejigging the critpath/bodhi/greenwave/openqa industrial complex (again) so we also/instead test everything that goes into the environments openQA tests (Server, Workstation, and KDE), even if it's not "important" in itself. Bad deps in *anything* in the environment can break the tests because they prevent `dnf update` from succeeding at all, so the updated packages never get installed (I actually think dnf-5 may be being more 'strict' than dnf here, too, and exposing problems that were previously not causing test failures, but I'd have to check that). This would be some fairly major surgery, though (and increase the load on openQA). -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@xxxxxxxxxxxxx https://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
