Hi all, as a Copr contributor, I am missing a _standard_ design for integrating our cool infrastructure into the _upstream_ work-flows. We have a lot of teams trying to implement the same thing: 1. Catch some upstream event (like push, pull-request, release, tag, ...) 2. React to the event. Do something with this event, (e.g. do CI build, Copr build, do some scratch build using Jenkins, etc.) 3. Ideally report back upstream, somehow, the results. By _standard_ I mean that (a) it should be absolutely trivial to start the Fedora Infra <-> Upstream interaction (to integrate) from the package contributor POV, and (b) that it should be _fairly simple_ to implement the reaction (point 2 and 3) inside our infra. Namely, consider that we in Copr want to trigger builds by the events from GitHub (PRs, tags, pushes, ...), GitLab (MRs, ...), Pagure, BitBucket, etc. We sort of are able to do so (it is far from complete support!), but for this to happen -- we had to implement pretty complicated webhook interface on Copr side for all the services, and for Pagure we have to listen on fedora-messaging bus. The real challenge though is _how to notify_ upstream projects that something inside our infra (eg Copr) happened. Nothing dramatic happened in Copr so far. We "only" have support for notifying Pagure ATM (we are able to set "commit status" for PRs and pushes). To be able to communicate with other forges, it would be an equivalent amount of repetitive work.. (we'd have to store somewhere the access "tokens", know how to/where contact the upstream, etc.). Quite some work, ... worth doing only once in whole Fedora? What I think we could have is something like a central gateway between Fedora and Upstream, let's say "Fedora Integrator"? That thing should cut out and provide the part of Copr code which does this integration steps: - It should "listen" on several forges (webhooks, on our bus, atm.), and translate the variously formatted event payloads into well-defined, uniform events. Say "change", "change request", etc. - Such an event would be dumped (back) on the fedora-messaging bus, through well-defined (schema) message. The bus part is important because it is (a) asynchronous and (b) even services behind corporate VPN can react on them. - Each such event would have its own ID, so **any** Fedora service can react on it by posting ReactsOn=EventID (again well-defined) "reaction" on fedora-message bus. - The Integrator service would be able to translate the "reactions" on bus to forge responses -- e.g., when CI passes, notify Pagure/GitHub/GitLab/... commit status, send an email to upstream, etc. I sketched a simple component graph how that could look like [1]. Feel free to comment here or there... the webhook/message payload translation mechanism should be somehow wrapped into a library, so we could still get the unified format of events without the integrator service (iff needed). This way we would build a single place for collaboration on such things (parsing the webhooks, forge interaction, etc.). But we would have a single orientation points for people who use our services for CI/CD (single "Fedora Integrator" GitHub app, GitLab.com "Fedora Integration", etc.). Not only that would be super easy to find -- such forge apps are also pretty privileged things (they can do various stuff with the repositories), so people might not be very confident they should enable them. If we had one blessed Fedora place to enable those things per forge, I think it would be more "trusted". Where to start? E.g. by using parts of github2fedmsg, copr and Packit service? We need to find a way to get the events on our bus easily, so I e.g. asked GitLab people [1] how to easily implement this. Do you think such initiative makes sense? [1] https://github.com/praiskup/fedora-integration-service [2] https://gitlab.com/gitlab-org/gitlab/-/issues/225151 Pavel On Thursday, June 25, 2020 9:27:24 PM CEST Pierre-Yves Chibon wrote: > Good Morning Everyone, > > Just like every team we have technical debt in our work. > I would like your help to try to define what it is for us. > > So far, I've come up with the following: > - python3 support/migration > - fedora-messaging > - fedora-messaging schema > - documentation > - (unit-)tests > - OpenID Connect > > What else would we want in there? > > > Looking forward to your thoughts, > Pierre > _______________________________________________ > infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx > _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
