On Wed, 2025-03-05 at 19:52 +0200, Jani Nikula wrote: > On Wed, 05 Mar 2025, "Knop, Ryszard" <ryszard.knop@xxxxxxxxx> wrote: > > Hey everyone, > > > > Patchwork has been having lots of issues recently, dropping patches, > > being unusably slow and generally starting to become more of a pain > > than help. Over on the CI side we are also not super fond of it and we > > don't have enough resources to maintain it properly. Lucas has > > suggested using public-inbox archives directly, and with some recent > > tools like b4/lei making common ML workflows feasible without > > reinventing the wheel, we are considering setting up a bridge between > > MLs and GitHub/GitLab to instead drive CI using more modern tools. > > I am supportive of this change. > > > We have not decided whether to drop Patchwork yet - this is a thread to > > gather people's thoughts if this sounds like a good idea. > > > > The workflow would look like this: > > > > - A drm-tip mirror would be set up on GitHub/fd.o GitLab, automatically > > pulling all changes from drm-tip upstream fd.o GitLab as a secondary > > source. > > - For each new series on lore.kernel.org a bridge would create a PR by > > taking the latest mirrored drm-tip source, then applying a new series > > with `b4 shazam`. > > There's a small catch here. Patchwork is currently more clever about > handling series revisions when only some of the patches in a series are > updated by way of replying to the individual patch. For example [1][2]. > > I've been meaning to report it to upstream b4, but haven't gotten around > to it yet. But I wouldn't consider this a blocker. > > [1] https://lore.kernel.org/r/20250305114820.3523077-2-imre.deak@xxxxxxxxx > [2] https://patchwork.freedesktop.org/series/145782/ > > > - That PR would then go through the normal CI flow, with CI checks > > being reported on that PR, instead of sending all the reports to the > > mailing list. > > - On the mailing list, the bridge would send an ack that a series has > > been seen and where are its results. You would no longer receive > > multiple emails with KBs of logs in your email client, but everything > > would be available from PR checks (as status checks and links to full > > logs only, no trimming and "last 1000 lines only"). > > \o/ > > > - Mirrors, PRs and checks for public mailing lists would be public, > > much like on the current public Patchwork instance. > > - Logs behind links will be stored for a few months (3-6, depends on > > traffic and how the situation evolves). GitHub Checks themselves (check > > status, shortlogs and links) have a hard retention period of 400 days. > > - Not sure about PR retention: we need a mechanism to correctly > > identify merged series somehow, then to trim these from the list. > > Expected retention time? > > With the velocity of the driver development, the test results go stale > in a matter of a week or two. I normally wouldn't merge patches older > than that without a fresh test round. > > IMO a long retention isn't necessary. At most a few months? The patches > will still stay on the list forever. > > > - Not sure about reporting on "CI finished": Maybe we could send one > > more email with a summary of checks when the entire workflow has been > > finished? > > I've previously suggested one short mail as an acknowledgement with a > URL to the results, and another mail when testing has ended one way or > another. I think it would be a good starting point. > > > On GitHub vs fd.o GitLab: I'm thinking more of GitHub here: > > > > - GitHub generally performs better with large repositories. > > - Extra fallback drm-tip source for fd.o downtime periods. > > - Bonus points: We can store public Intel CI tags directly in that > > mirror for moderate periods of time without abusing fd.o servers. > > > > Either option would work fine though, so opinions here would be > > appreciated :) > > I think eventually we will want to consider accepting contributions via > gitlab merge requests directly. > > It would also be interesting if maintainers/committers could merge the > contributions via gitlab UI already when CI applied the patches from the > mailing list and created the merge request. > > In the merge request case, they'd have to be against individual repos > that feed into drm-tip, *not* merge requests against drm-tip > directly. So for testing CI would have to recreate drm-tip the same way > as 'dim push-branch' currently does. This is doable, but perf-wise is not going to be great. We would have to checkout all trees pulled into drm/tip for each build as listed in the latest integration-manifest, replace target tree with the MR tree, then provide results from that. We'll see how this works out in practice. (It should be just `dim rebuild-tip` after pointing all the branches at the required commits?) This also means having a backup drm/tip source when fd.o is offline is out; it's patched into too many places if dim gets used. Ryszard > I don't think these are hard requests at this time, and should not block > the forward progress, but it's maybe something you want to consider so > you're not inadvertently creating problems for your future self! > > > CI itself would not run in the native forge CI either way, but rather > > on our Jenkins infra, as it does today. If there's ever a need to > > switch forges, it would require reworking the bridging/reporting bits, > > but not *all* the bits. > > > > We don't want to self-host another forge as we don't have enough people > > and free time to do that properly. Codeberg, etc are not an option due > > to the drm-tip repo size. > > > > And that's the initial idea. Thoughts? Do you like any of this, or does > > it sound like a downgrade from what we have today? > > I think it sounds good overall. I don't like the flood of mails, and > they don't have complete information anyway. I'm hopeful using > github/gitlab would make the whole CI slightly more transparent too. > > I wouldn't mind sunsetting fdo patchwork at all. > > > BR, > Jani. >
