Summary: I think we can fix the ELN side-tag rebuild problems and make the composes more reliable if we change the mechanism for kicking off rebuilds. I'm soliciting feedback to help identify potential issues with this proposed approach. ## Background Information ## Currently in ELN, merging a side-tag into Rawhide results in all of the packages from that side-tag being rebuilt concurrently in ELN. This leads to two problems: 1. Side-tags containing large numbers of package builds will trigger many ELN builds at the same time, possibly overwhelming available resources on the ELN automation systems. 2. Many (most?) side-tags exist to ensure that packages are built in a particular order so as to ensure that they are built after their dependencies. Launching all the rebuilds concurrently means that many of the builds may succeed *and still be wrong* (such as if they are built against an older soname). ## Proposed Solution ## I had a discussion with Miro Hrončok this morning where we tackled this problem and may have come up with a workable solution for 99% of cases. Instead of treating side-tags as a special event and trying to sort the builds such that they are built in the same order, we can instead tag in the Rawhide packages first, then issue the rebuilds together. With the Rawhide packages available, we won't need to worry about the ordering, because the dependencies will already be present in a sufficiently-recent version. As a bonus, we'll reduce the likelihood of broken ELN composes, since if an ELN rebuild fails, the Rawhide version will still be present to satisfy dependencies. In greater detail: Whenever a build is tagged into the 'f35' tag (later, whatever tag matches Rawhide), ELN automation would take the following steps: * Identify whether this package is on the list of packages that ELN rebuilds[1] * Tag the Rawhide build into the 'eln' tag (so it is now tagged with both 'f35' and 'eln') * Enqueue a Koji build against the 'eln' target from the same Git commit The queue mentioned above should be maintained in a separate thread and used to submit tasks in batches to avoid overloading the infrastructure. If the Koji build against the 'eln' target fails, the Rawhide build will remain as the most-recently-tagged version of the package in ELN and become part of the compose until the ELN rebuild can be fixed. Note that this process would apply to ALL builds in Rawhide, not just those coming from side-tags. There would be no difference in behavior between standard direct builds and side-tag merged builds. ## Known potential issues ## * Some packages may auto-detect functionality based on functionality made available by one of their dependencies. If the Rawhide and ELN versions of that dependency differ in visible functionality, then building an ELN package with a Rawhide version of its dependency could result in unexpected behavior. I believe this issue to be rare and generally best handled by the packager as the subject matter expert. They'd just need to bump the release number and rebuild the package in ELN. Alternatively, if this is known to be regularly problematic for a package, the maintainer can opt out of the automatic rebuild and work out a strategy with the ELN SIG for dealing with it. [1] This will be the set of packages provided by https://tiny.distro.builders/view--view-eln.html minus any packages that have opted out of automatic rebuilds (they perform manual rebuilds for ELN). _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
