Rawhide update gating: mass rebuild issues

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


Hi folks! Just wanted to drop a note for anyone who noticed their
Rawhide updates stuck in gating for an unusually long time

This is mainly due to the mass rebuild. Merging it in (which was done
without sending anything in it through openQA testing) resulted in two
separate problems:

1. Some GNOME packages were inadvertently bumped to 45-alpha early, as
the changes were checked into git but had not been built; this caused
dependency issues, then when we tried building the remaining packages
at version 45-alpha to see if things would work OK with a consistent
set of packages, we found a bug which made the tests fail and
necessitated untagging everything back to 44:
https://gitlab.gnome.org/GNOME/mutter/-/issues/2918 .

2. The rebuilt PackageKit was broken because of a change in glib, and
needed a backport of https://github.com/PackageKit/PackageKit/pull/643
to work properly again.

These were complicated by a third problem showing up in the middle: a
new build of brltty had some internal dependency issues, and needed to
be untagged.

All of these issues caused test failures for *every* Rawhide update
tested after the mass rebuild was merged (in the first case) or the
brltty update landed (in the third case). It took several hours to get
them all unpicked and the fixed PackageKit landed, then I re-scheduled
the failed tests on all other updates (but I did this a bit wrong for
some of them, so they failed again, and I had to re-run them again an
hour or so ago).

I'm also fighting a bit with Koji giving 404 responses for the f39-
build repo more often than it usually does, but I hope to have tests
for all updates passing (assuming the update itself has no problems)
within the next hour or two.

For the next go-round, it might be good to use the relatively new
ability we have to schedule openQA tests on a side tag to test the mass
rebuild *before* it's merged, I'll check with releng if that is

This is also the second or third time gating has been disrupted by a
broken update that wasn't in the critical path and so didn't go through
gating itself. I'm considering rejigging the
critpath/bodhi/greenwave/openqa industrial complex (again) so we
also/instead test everything that goes into the environments openQA
tests (Server, Workstation, and KDE), even if it's not "important" in
itself. Bad deps in *anything* in the environment can break the tests
because they prevent `dnf update` from succeeding at all, so the
updated packages never get installed (I actually think dnf-5 may be
being more 'strict' than dnf here, too, and exposing problems that were
previously not causing test failures, but I'd have to check that). This
would be some fairly major surgery, though (and increase the load on
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@xxxxxxxxxxxxx

test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Index of Archives]     [Fedora Desktop]     [Fedora SELinux]     [Photo Sharing]     [Yosemite Forum]     [KDE Users]

  Powered by Linux