This idea sounds pretty good to me. As a less active contributor: How can I help out? Do you have a task list? (Sorry if this got already answered in one of the follow up emails.) Cheers, Dan Igor Gnatenko <ignatenkobrain@xxxxxxxxxxxxxxxxx> writes: > MayBe I …(can do something useful)? > > Hello, > > We've been discussing some (hopefully) nice idea with Mikolaj, Neal and > Jakub how we could improve packager (and user) experience and we have some > proposal which will be described below. > > We would like to ask you to read it, understand it and ask us any questions > you have. From the Fedora Infrastructure we would like to ask for some > machines to implement this idea (you can find some more information in > "Implementation details" part). > > I would like to apologize for HTML email, but it is much easier to have it > that way to have better visibility and reading easiness. > > Feel free to reply to this email or comment in google doc (there is a link > on the bottom). > Proposal Owners > > - > > Mikolaj Izdebski (mizdebsk) <mizdebsk@xxxxxxxxxx> - Java SIG, Fedora > infrastructure > - > > Igor Gnatenko (ignatenkobrain) <ignatenkobrain@xxxxxxxxxxxxxxxxx> - Rust > SIG, Golang SIG, Neuro SIG, RPM and libsolv contributor > - > > Neal Gompa (ngompa) <ngompa13@xxxxxxxxx> - Rust SIG, Golang SIG, RPM > contributor > - > > Jakub Čajka (jcajka) <jcajka@xxxxxxxxxxxxxxxxx> - Golang SIG, Container > SIG > > > This proposal aligns with the objective of improving the packager experience > <https://fedoraproject.org/wiki/Objectives/Packager_Experience> by > developing a platform whereby the proposal owners and others can experiment > with improvements that can be funneled back into the official production > infrastructure. > ProblemsProblem №1: Build-only packages > > Some ecosystems have many build-only packages (packages which are used to > build other packages, without having them installed on end-user systems). > Those ecosystems include Java, Rust and Go. > > It is extremely hard to keep up maintaining them due to lack of > time/people. Upstreams are usually changing quickly (sometimes when you > update just one such package, you’d need to update tens of the > dependencies). Few facts about such packages: > > - > > They are almost always outdated in released versions of distribution; > - > > They are often FTBFS (e.g. because there was compiler update but not > package update, broken deps, broken APIs due to deps rebases, …). > > > In Rust ecosystem, we abuse Rawhide to build and store such packages there > and then generate end-user application (e.g. ripgrep) which uses some of > those packages and produces binary for all supported releases. Those > applications have high quality and are supported. > > Rawhide gating makes this much more complicated because builds appear in > buildroot slower, updating group of packages would need side tags and it’s > just painful to work with. > > https://pagure.io/fesco/issue/2068 > > And, after all, those packages shouldn’t be shipped to users. > Problem №2: Testing of new rpm/koji/mock features/configuration > > When developing new features in RPM (e.g. rich dependencies) or testing > different Koji configuration (e.g. removing gcc/gcc-c++ from the > buildroot), it is required to make custom configuration and try building > things. > Problem №3: Developing modules > > Modules are built in MBS as a single unit. It is hard to develop big > modules by progressively improving individual packages or small package > groups. Scratch builds for modules are not implemented. Local builds work > differently from official module builds, they don’t scale and don’t allow > groups of people to work collaboratively. All these problems can be solved > by first developing a flat repository of packages in a shared environment > and then generating modulemd from such package set. > Problem №4: Testing things before push > > Primary Fedora Koji and dist-git are not suited for packaging > experimentation. Packagers are expected to experiment on their own systems > and push changes of successful experiments only. But this approach doesn’t > scale with number of maintained packages. Often you can find commits like > “d’oh, forgot to upload sources” or “let’s try with this settings”. People > need to build things somewhere quickly, do development and testing. And > only after that, they should push commit(s) to Fedora. > Solution > > - > > Separate Koji + Koschei deployed in Fedora infrastructure cloud; > - > > Builders are optimized for the best performance (see below > <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwGGgqJ_Y/edit#heading=h.amk0udnj85tg> > ); > - > > People can have their own targets where they can break things as they > wish without affecting others; > - > > Package changes are eventually contributed to Fedora proper by merging > changes to Fedora dist-git and rebuilding packages in official Fedora Koji; > - > > All improvements can eventually be contributed back to official Fedora > infra. > > Ideas > > All ideas which you’ll find below are just an ideas and do not have to be > implemented. > Idea №1: Automated legal review > > openSUSE released their review app called Cavil > <https://github.com/openSUSE/cavil> which we could try running in automated > way. > Idea №2: Automated package review > > Currently review process is burden and we could run automated legal review, > create a repo, run set of checks and notify person. Somewhat similar to > fresque <https://github.com/fedora-infra/fresque>. In future might be > integrated with approval process and auto-imported into Fedora. > Idea №3: Building packages for multiple distributions > > In Rust ecosystem, we managed to get have same packaging guidelines in > openSUSE, Mageia and Fedora. We could automatically build some packages on > multiple distributions and be kinda upstream. > Idea №4: Building custom images with user content > > Deploy (or build) a tool(s) to enable user-built install, appliance and > container images with their content (modulo restrictions similar to COPR) > on top of Fedora. > Implementation details > > - > > Koji and Koschei deployed in fedorainfracloud.org > - > > A few builders constantly running, with a possibility to add more > builders as needed and as available cloud resources allow > - > > Deployed and maintained by proposal owners - not supported by Fedora > infrastructure > - > > Other contributors can have access granted by proposal owners (for the > time being only users in “packagers” group will be eligible to get access) > - > > Possibility to have some builders running outsides of Fedora > infrastructure - contributed by proposal owners > - > > Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or > from SRPM upload > > FAQWhy not COPR? > > - > > COPR has been starved of resources for years, which has impaired its > ability to provide reliable service at scale. Fedora Infrastructure refuses > to consider supporting it officially and Fedora Release Engineering seems > to have some undefined issues with COPR. > - > > It is not official build system of Fedora which is not helping with Problem > №2: Testing of new rpm/koji/mock features/configuration > <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwGGgqJ_Y/edit#heading=h.iodoq4xw80c2> > . > - > > COPR is not extensible enough, more specifically: > - > > No query API (e.g. it is not possible to find out from which SCM > commit the package has been built) > - > > Builds always have access to all previous builds in a repository > (i.e. not possible to control when/how repo is created) > - > > GC doesn’t exist > - > > No scratch builds > > Why not staging Koji? > > - > > Staging Koji is meant for testing Koji itself and not for package > development. > - > > Staging Koji is often in broken state where it is not possible to build > anything. > - > > No one can touch staging Koji, so it’s pointless for offering a useful > service. > > How can you make Koji builds faster? > > - > > By tuning Koji for performance, not correctness and reliability. > - > > By building only on a single, fast architecture. > - > > By extensive caching. Files like RPM packages, repodata and files in > lookaside cache don’t change after they are initially stored, so builders > can cache them aggressively. > - > > By keeping build repositories small. Some package sets don’t need to be > built against full Fedora, but can use a minimal subset of Fedora. Such > repositories can be generated by Koji in seconds, not minutes. > > Why not the Open Build Service (OBS)? > > - > > OBS is not yet packaged for Fedora officially. The upstream code lacks > adaptations necessary to deploy and run on Fedora and in Fedora > Infrastructure. > - > > OBS is not official build system of Fedora, which does not help with Problem > №2: Testing of new rpm/koji/mock features/configuration > <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwGGgqJ_Y/edit#heading=h.iodoq4xw80c2> > . > > What does MBI stand for? > > - > > M for middlestream, module, mizdebsk, … > - > > B for build, bootstrap, … > - > > I for infrastructure, initiative, ignatenkobrain, … > - > > MBI might be pronounced as “maybe I …(can make something cool)?” > - > > MBI is also initials of mizdebsk (Mikolaj Bozydar Izdebski) > - > > MBI is also IBM spelled backwards :) > > Is it somehow related to Fedora Playground > <https://fedoraproject.org/wiki/Playground>? > > Yes, it is. Although it is more developer-focused. Users could enrol for > some specific things like experimental Java/Rust packages. > > > MBI (Playground 2.0) > <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwGGgqJ_Y/edit?usp=drive_web> > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
