F43 Change Proposal: Packit as a dist-git CI (system-wide)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Wiki - https://fedoraproject.org/wiki/Changes/PackitDistgitCI
Discussion Thread -
https://discussion.fedoraproject.org/t/f43-change-proposal-packit-as-a-dist-git-ci-system-wide/140619

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
This change aims to change a dist-git CI solution to one based on
Packit and deprecate current solutions (Fedora CI and Zuul).

This change does not affect the tests being run and the test execution
service (=Testing Farm), but the mechanism used to trigger and report
them.

This is strictly not related to the opt-in Packit-provided workflows.
(I.e. this does not require using Packit for release syncing.) A
slightly different branding might be used to avoid confusion.


== Owner ==

* Name: [[User:lachmanfrantisek| František Lachman]]
* Email: flachman@xxxxxxxxxx

* Name: [[User:lbarczio| Laura Barcziova]]
* Email: lbarczio@xxxxxxxxxx

* Name: [[User:mmassari| Maja Massarini]]
* Email: mmassari@xxxxxxxxxx

* Name: [[User:mfocko| Matej Focko]]
* Email: mfocko@xxxxxxxxxx

* Name: [[User:nforro| Nikola Forro]]
* Email: nforro@xxxxxxxxxx




== Detailed Description ==
Since Packit has a long-term and feature-rich integration with Testing
Farm and also automation around dist-git pull requests, the Packit
team is willing to help with a current, non-ideal, situation with
dist-git CI. It looks like there are not enough resources to properly
support and improve the current solutions (Fedora CI and Zuul). This
proposal does not influence Fedora CI Bodhi testing but might be
considered as a next step.

Fedora CI maintenance had been a long-term concern without a clear
owner for the dist-git integration and with plans to move it to the
Testing Farm team. The Testing Farm team would like to avoid
maintenance of these and rather have only a single integrator both
upstream and downstream.

Since Packit Fedora automation heavily relies on working CI on
dist-git pull requests, we need to make sure it works reliably.

Most of the code is already there in
[https://github.com/packit/packit-service/ Packit’s codebase]. The
work is mainly about plumbing everything together and making sure it
works as expected.

We welcome any suggestions but these are the phases we are thinking about:
* '''phase 0: (current state)'''
** opt-in (by Packit service configuration)
** Scratch build being run (example PR)
* '''phase 1:'''
** opt-in (publicly available)
** Scratch build being run
** time: ~ month
* '''phase 2:'''
** opt-in (publicly available)
** scratch build +
[https://github.com/fedora-ci/installability-pipeline installability
check]
** time: ~ month
* '''phase 3:'''
** opt-in (publicly available)
** scratch build +
[https://github.com/fedora-ci/installability-pipeline installability
check] + user-defined TMT tests (as separate results)
** time: ~ month
* '''final phase ~ Fedora CI replacement:'''
**  by-default
** Fedora CI is deprecated
** Separate Packit deployment instance for CI; running in the Fedora
Infrastructure.
** opt-out in favour of Zuul
** scratch build +
[https://github.com/fedora-ci/installability-pipeline installability
check] + user-defined TMT tests (as separate results)
** time: ~ enough time for people to try before defaulting
* '''next steps'''
** Everything currently run by Fedora Zuul on dist-git pull requests
is run via Packit (opt-in).
** Deprecate dist-git tenant on Fedora Zuul instance.

=== Risks: ===
* Deployment to Fedora infrastructure
** Despite Packit requiring only OpenShift for its deployment, there
can still be problems found when deploying to another cluster.
** Some work is required to be done by the Fedora Infrastructure team
that we can’t influence.
* Performance
** When going to the final phase when run by default, the load might
significantly increase. Despite the number of packages actively using
dist-git pull requests is not huge, we still need to be prepared.
* dist-git git forge switch
** Despite Packit being written in a git-forge-independent way, there
can still be some issues related to it (be it on Packit's side or
not).
** (Forgejo-support in the underlying forge-unifying library is being
researched and [https://github.com/packit/ogr/issues/866 work] can
start soon.)

=== Support: ===
* For transparency, the Packit team is a Red Hat team under the same
manager as the CPT team (=Copr and Mock) and in the same group as
Testing Farm and other community-related projects.
* The team works and plans in public (see
[https://github.com/orgs/packit/projects/7 the Packit Team Kanban
Board]) and welcomes any external contributor to collaborate.
* To have more Fedora control over the suggested service, the team
wants to deploy it to the Fedora Infrastructure OpenShift cluster and
give the Fedora Infrastructure team access to it so the Packit team
can’t become a bottleneck.
* In case of interest, we welcome anyone to join us in this effort.
* We are active in the #packit@xxxxxxxxx Matrix channel but a separate
channel can be used to improve the communication.

A couple of links to better understand the current situation:
* https://pagure.io/fedora-ci/general/issue/476
* https://pagure.io/fedora-infrastructure/issue/11920
* https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/F4G35DWMCNMOYA74PZ7SE7VFPUZOTAZ5/
* https://github.com/packit/packit-service/issues/2453

== Feedback ==
* Confusion with the current Packit workflows
** Rebranding might be needed.
* There are concerns about fragmentation across Fedora services.
** Packit’s goal is not to implement everything but rather integrate
existing services and provide the best user experience. This can lead
to the reduction of services users need to interact with.
** Packit is already in the ecosystem so it’s about using it for more use cases.
* STI is currently not supported by Packit.
** Technically it can be introduced (because Testing Farm supports
it), but since it has been obsoleted for some time and there is a
[https://fedoraproject.org/wiki/Changes/DeprecateSTI Change Proposal]
to not support it, this might not be needed at all. (We can use this
as an opportunity to move away.)
* What to do about test/pipeline definition repositories? Who should
be responsible for it and how to better collaborate on these?
** As a result of this change, we would like to improve also situation
around the responsibilities of shared plans and clearly state who is
responsible for which part. Packit can but not need to maintain these
repositories.
* Should Packit aim to run Zuul plans for everyone? (The plans that
are not run by Fedora CI.)
** Ideally, the topic of the check choice should not be part of this
change but we can do the installability check by default and let
people opt-in to checks currently run by Zuul. (And make it possible
to run these by default if there is a broader agreement.)
* Should there be an opt-out mechanism for the final solution?
** If needed, we can allow this, but ideally, there should be a
broader discussion and agreement if packages can opt-out from default
jobs.
* Why not pick current solutions?
** Covered more in the Benefits section and motivation, but the main
difference is not so much in technical differences but in a team
dedicated to maintaining and further developing the solution.
* What will be the effect on Koji? Won't it increase the load because
of the duplicate scratch builds?
** At the end, there will be only a single dist-git CI so there won't
be any duplicates.
** For the opt-in phases, the duplicates are possible.
*** We can ask people to opt-out from Zuul if they opt-in to
Packit-based CI. For Fedora CI, we would welcome any suggestion if
there isn't a way to opt-out from it.
*** We can also try to share scratch builds with other solutions but
since it's only for a limited time, this might not be worth the time.

== Benefit to Fedora ==
* Single CI system to run and maintain.
** Both for the user and resource efficiency (e.g. single scratch
build, single service to maintain).
* Reliable and actively maintained+developed CI
** Most importantly, the Packit team offers not only the
implementation but also people to support the service. (We consider
this to be the main issue with the current solutions.)
** We sadly hear regular complaints about the stability of both
existing solutions and issues needing to be fixed for a long time.
(This is mainly related to the previous point that people do not have
enough time for it.)
** Packit has a history of being reliable
([https://packit.dev/docs/service-level-objectives#slo1-changes-to-github-prs-receive-the-first-status-update-within-15-seconds-in-99-of-cases
current set of SLOs], dist-git specific ones might be defined) and has
a dedicated team.
** We also developed multiple mechanisms to make the testing process
more reliable (e.g. auto-retry on infrastructure issues and so-called
babysitting tasks to check for results if the result message got
lost).
* Feature-rich CI experience
** The CI integrations do not need to be only about triggering the
test job and reporting, but also about being able to rerun a
particular part of the test suite or influence the test job workflow
based on some rules (e.g. based on labels). People can decide what to
run and if to run it automatically or manually. This is especially
useful when speaking about user-defined tests.
* Same UX for upstream and downstream
** This would allow people to have the very same test experience and
definitions both upstream and downstream.
** Packit handles not only tests. We want to unify the experience for
the whole pipeline from upstream through dist-git and Koji to Bodhi.
* Implementation of git-forge agnostic thanks to a unified git-forge
API (Pagure and GitLab are available now, Forgejo is being researched
and [https://github.com/packit/ogr/issues/866 work] can start soon).
** This work can be coordinated with the git-forge swap for a smoother
transition and to save time migrating the existing solutions.
* Possible future enhancements when this work is done:
** Copr builds (available in upstream)
** OpenScanHub checks (available in upstream)
** Reverse-dependency builds/tests (via Koschei, research done)

== Scope ==
* Proposal owners:
** Implement the solution following the phases above and communicate
the milestones publically.
** Update Fedora documentation.
** Agree on the ownership of shared test plans.

* Other developers:
No functional change that might affect maintainers. Only UX differences.

* Release engineering: -
** Coordination with release engineering is not required. A mass
rebuild is not necessary for this change.

* Policies and guidelines: documentation needs to be updated

* Trademark approval: N/A (not needed for this Change)

* Alignment with the Fedora Strategy:
This proposal aligns with Fedora’s strategy of improving contributor
experience and easing workflows, supporting a more efficient and
user-friendly ecosystem.


== Upgrade/compatibility impact ==


== Early Testing (Optional) ==
Do you require 'QA Blueprint' support? N

== How To Test ==
During phases 1-3, one can opt-in to this solution and try it for
real. (Most probably via global configuration, similar to the Zuul
approach. Team has a
[https://github.com/packit/packit-service/issues/2628 research task]
in progress on this topic. Suggestions welcome.)


== User Experience ==
* The functionality and workflow will be preserved.
* As a reaction to a dist-git pull request a scratch build and a
follow-up test are being triggered. Users are notified about progress
via statuses that lead to [https://dashboard.packit.dev/ Packit
dashboard] providing more details about builds and tests.


== Dependencies ==
Ending support for STI tests (being submitted as another change) might
ease the implementation.


== Contingency Plan ==
* Contingency mechanism: (What to do?  Who will do it?) N/A
* Contingency deadline: N/A
* Blocks release? No

If the implementation is not ready in time, we can (maybe temporarily)
stay with one of the current solutions.


== Documentation ==
Team research about technical feasibility of this functionality:
https://packit.dev/research/integrations/fedora-ci


== Release Notes ==


-- 
Aoife Moloney

Fedora Operations Architect

Fedora Project

Matrix: @amoloney:fedora.im

IRC: amoloney

-- 
_______________________________________________
devel-announce mailing list -- devel-announce@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-announce-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- 
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux