https://fedoraproject.org/wiki/Changes/GatingRawhidePackages Note that this was previously submitted as "Gating Rawhide - Single package updates": https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/SBB3DJO27HDDO6NAW54CLH2M62FEG622/ == Summary == We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working. == Owner == * Name: [[User:pingou|Pierre-Yves Chibon]] * Email: pingou - pingoured.fr People who are/will be involved in this: * Coordinator: [[User:pingou|Pierre-Yves Chibon]] * Bodhi: [[User:bowlofeggs|Randy Barlow]], [[User:abompard|Aurélien Bompard]] and [[User:cverna|Clément Verna]] * koji: [[User:mizdebsk|Mikolaj Izdebski]] and [[User:mikem|Mike McLean]] * fedpkg/rpkg: [[User:cqi|Chenxiong ]] and [[User:lsedlar|Lubomír Sedlář]] * Fedora CI: [[User:dperpeet|Dominik Perpeet]] == Detailed Description == Rawhide is an unique place in the Fedora eco-system. It is the only place where large changes and rebuilds can be done. In stable Fedora releases, the rebuilds of large sets of packages is discouraged by the packaging guidelines, so the ratio of number of updates in bodhi which a single build vs multiple builds (95% to 5%) is not reflective of the reality of the rawhide ecosystem. The basic principal of this proposal is to provide an environment in which packages can be built and tested without affecting other packages. When considering gating rawhide package updates on test results, we need to consider two workflows: single build updates, multi-builds updates. For single build, the easiest solution is to provide a dedicated koji tag in which these packages are built and where they will wait for their tests to pass before they can enter the buildroot. For multi-builds, the solution will be to rely on side-tags in koji. These side-tags are basically tags created by the packager and available to them to do their work, in this case rebuild all the packages desired. The side-tag can then sent to the test system as being one unit of change. To use a, now well-known, analogy, a single-build update is like sending a commit to a mailing list, it waits there to be reviewed and tested before being merged into the main repository. Multi-build updates are like pull-requests, they can contain one or more builds and which are reviewed and tested all together as one change before being merged. Within this proposal, we aim at building the infrastructure allowing to gate packages in rawhide. The goal is for packages to go through bodhi, be tested and if the tests pass, land in the rawhide buildroot as they do today. In the simplest case, the packager workflow will not be affected by this proposal, less simple situation will require adjustements to the packager workflow that we would like to try to be as minimal as possible. We do not aim at making any tests mandatory as part of these proposals. Once packages gating is in place it will be up to the community and likely FESCo to decide if they would like to enforce some rules on all packages and which ones. Note that this proposal completes previous initiative such as [[Changes/NoMoreAlpha]]. === Workflows === <gallery> Single_package_GatingRawhide_bodhi.png|Single package update with gating GatingRawhide_MultiplePkgs_bodhi.png|Multi-packages update with gating </gallery> == Benefit to Fedora == * As more packagers opt-into gating and add tests to their packages, rawhide becomes more stable * More stable rawhide will lead to easier composes and a smoother release process * Ability to use chain-build in rawhide and stable releases == Scope == The scope of this work is adding a mechanism to gate builds/updates before they enter the rawhide buildroot (and thus become accessible to others) so broken packages are kept out and cannot affect other packages or the compose until they are fixed by their maintainers. '''The CI system, the tests and the decision on which tests are used to gate upon are out of scope for the present document.''' * Proposal owners: pingou will be coordinating the work among the different stack holders * Other developers: ** Bodhi developers: Implement the needed changes ** koji developers: Implement the needed changes ** fedpkg/rpkg developers: Implement the needed changes ** infrastructure: deploy the new version of bodhi and koji * Release engineering: [https://pagure.io/releng/issue/8183 #8183] * Policies and guidelines: Once the entire system is in place, FESCo may want to set a policy on distribution-wide gating (ie: tests that would be enforced for all packages in the distributions). This is however out of scope of this proposal. * Trademark approval: N/A (not needed for this Change) === Application impacted === ==== Bodhi ==== * A message-bus listener will automatically create a bodhi update for each package built in the rawhide-gated tag. * A message-bus listener will automatically push bodhi updates created for rawhide that have passed CI (the decision being announced by Greenwave). * A message-bus listener will automatically comment on bodhi update about update in greenwave's decision ** This will notify users about the test results and the corresponding gating status. * bodhi will support creating a new update from a side-tag instead of a list of builds. ==== Koji ==== * A koji plugin will allow packagers to create new side-tags * How side-tags are created will be optimized so they do not take hours to be available to the user requesting them ==== fedpkg/rpkg ==== * The command: `fedpkg side-tag create` will be added to create a new side-tag in koji for this user using the new plugin * The command: `fedpkg side-tag merge` will be added to create a bodhi update corresponding from this side-tag ** Alternatively: Expand the existing `fedpkg update` command to add support for creating a new update from a side-tag ==== Greenwave / WaiverDB ==== Nothing should change for these tools but we will have to confirm this and test in staging that they behave as expected. ==== CI system ==== Nothing should change for the CI system but we will have to confirm this and test in staging that they behave as expected. == Upgrade/compatibility impact == N/A == How To Test == === Single build === ==== With opting in into gating ==== # Add tests to a package you maintain in Fedora using the [https://docs.fedoraproject.org/en-US/ci/quick-start-guide/ ci/quick-start-guide] documentation (see the full [https://docs.fedoraproject.org/en-US/ci/ CI documentation] for more information). # Update your package in rawhide # Ensure that the bodhi update is automatically created # Check that the tests have properly ran (see the Tests tab in the bodhi update) # Ensure that the bodhi update went through to rawhide once the tests passed If one or more tests failed and you wish to waive them # Waive the failing tests using `bodhi updates waive <id>` see `bodhi updates waive --help` for more information ==== Without opting in into gating ==== # Ensure your package does not have a `gating.yaml` in its dist-git repository # Build your package in rawhide as you do today # Ensure that the bodhi update is automatically created # Ensure that the bodhi update goes through to rawhide automatically === Multi-builds === # Add tests to a package you maintain in Fedora using the [https://docs.fedoraproject.org/en-US/ci/quick-start-guide/ ci/quick-start-guide] documentation (see the full [https://docs.fedoraproject.org/en-US/ci/ CI documentation] for more information). # Request a new side-tag in koji for your rebuilds # Update your packages in this side-tag # Request for this side-tag to be merged either using fedpkg or bodhi's UI (create a new update from a side-tag) # Check that the tests have properly ran (see the Tests tab in the bodhi update) # Ensure that the bodhi update went through to rawhide once the tests passed # Check that the side-tag was correctly cleaned afterward ==== Without opting in into gating ==== # Ensure all your packages do not have a `gating.yaml` in its dist-git repository # Build your packages in rawhide as you do today # Ensure that the bodhi updates are automatically created # Ensure that the bodhi updates go through to rawhide automatically == User Experience == === Musts === Here below are the list of requirements for this proposal: * Packagers must be notified when the update corresponding to their build(s) passes or fails CI * Packagers must be able to see what is blocking an update (ie missing requirements/test results) * Packagers must be able to re-trigger a test run (via either an UI or a CLI) * Packagers must be able to waive false-negative (via either an UI or a CLI) === Nice to have === Here below is a list of ideas that would make the user experience more friendly but will not be part of the initial release: * Packagers would like to follow the progress of the tests ** This could be implemented by notifying the packagers when a test run starts and provide an URL where they could see and follow them * Packagers would like to be able to re-trigger a test run from an UI and a CLI * Packagers would like to be able to waive false-negative from an UI and a CLI * Packagers would like the ability to transform a bodhi update automatically created for a single build into a side-tag-specifi bodhi update ** This would allow the workflow where a packager makes a build, which fails CI because another package needs to be rebuilt as well. The packager can then convert the bodhi update into a side-tag based update, this would create the corresponding side-tag and move the existing build into it, allowing the packager to do the missing rebuilds before telling bodhi to process the update. This is entirely doable by hand, which is why this feature while convenient is considered a nice to have * Packagers would like fedpkg chain-build to create the corresponding side-tag automatically === Single build update === fedpkg clone rpms/foobar cd foobar/ vim foobar.spec git commit -am "Git commit message" fedpkg build If the CI tests pass, the package will land in rawhide, if they fail the user will be able to go to bodhi to see what failed and why as well as waiving the failed tests if needed. === Multi builds with gating === fedpkg clone rpms/foobar cd foobar/ fedpkg side-tag create foo1.2 vim foobar.spec git commit -am "Update of foobar to update foo to 1.2" fedpkg push fedpkg build --target=user_foo1.2 cd .. fedpkg clone rpms/foobaz cd foobaz vim foobaz.spec git commit -am "Update of foobaz to update foo to 1.2" fedpkg push fedpkg build --target=user_foo1.2 cd .. fedpkg clone rpms/foo cd foo vim foo.spec git commit -am "Update foo to 1.2" fedpkg push fedpkg build --target=user_foo1.2 fedpkg side-tag merge foo1.2 == Dependencies == * Bodhi * koji * fedpkg == Contingency Plan == We can keep doing rawhide as we are doing it today. == Documentation == To be written/updated List of documentation pages to update: * https://fedoraproject.org/wiki/Updates_Policy * https://fedoraproject.org/wiki/Package_maintenance_guide * https://fedoraproject.org/wiki/Package_update_HOWTO * https://docs.fedoraproject.org/en-US/packaging-guidelines/ -- Ben Cotton Fedora Program Manager TZ=America/Indiana/Indianapolis Pronouns: he/him _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
