On 1/31/19 4:24 AM, Igor Gnatenko wrote: > ProblemsProblem №1: Build-only packages > > Some ecosystems have many build-only packages (packages which are used to > build other packages, without having them installed on end-user systems). > Those ecosystems include Java, Rust and Go. > > It is extremely hard to keep up maintaining them due to lack of > time/people. Upstreams are usually changing quickly (sometimes when you > update just one such package, you’d need to update tens of the > dependencies). Few facts about such packages: > > - > > They are almost always outdated in released versions of distribution; > - > > They are often FTBFS (e.g. because there was compiler update but not > package update, broken deps, broken APIs due to deps rebases, …). > > > In Rust ecosystem, we abuse Rawhide to build and store such packages there > and then generate end-user application (e.g. ripgrep) which uses some of > those packages and produces binary for all supported releases. Those > applications have high quality and are supported. > > Rawhide gating makes this much more complicated because builds appear in > buildroot slower, updating group of packages would need side tags and it’s > just painful to work with. Note that the rawhide gating plans include self service side tags, so you can make one anytime you want to update a collection of packages. This of course doesn't solve the above problem, but it's worth mentioning I think. > https://pagure.io/fesco/issue/2068 > > And, after all, those packages shouldn’t be shipped to users. This of course means however that users who want to build their own verions of the applications or whatever can't start from our repos, they have to use whatever setup we use for developing them. I don't know any answers here but it seems to me we keep making it harder for people to use Fedora to develop applications. ...snip... > Solution > > - > > Separate Koji + Koschei deployed in Fedora infrastructure cloud; Turns out we are going to be retiring our cloud, so no, this is not a place to put it. :) > - > > Builders are optimized for the best performance (see below > <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwGGgqJ_Y/edit#heading=h.amk0udnj85tg> > ); > - > > People can have their own targets where they can break things as they > wish without affecting others; Would that be entirely self service? Or ? > - > > Package changes are eventually contributed to Fedora proper by merging > changes to Fedora dist-git and rebuilding packages in official Fedora Koji; Where is the source for the changes from? Would this need it's own pagure/src/dist-git? Or would it pull from PR's in prod dist git? Or ? > - > > All improvements can eventually be contributed back to official Fedora > infra. Sounds great! > > Ideas > > All ideas which you’ll find below are just an ideas and do not have to be > implemented. > Idea №1: Automated legal review > > openSUSE released their review app called Cavil > <https://github.com/openSUSE/cavil> which we could try running in automated > way. I guess this would be up to legal to look at and tell us if it's worth doing. > Idea №2: Automated package review > > Currently review process is burden and we could run automated legal review, > create a repo, run set of checks and notify person. Somewhat similar to > fresque <https://github.com/fedora-infra/fresque>. In future might be > integrated with approval process and auto-imported into Fedora. Sounds great if someone has time to build the app(s) > Idea №3: Building packages for multiple distributions > > In Rust ecosystem, we managed to get have same packaging guidelines in > openSUSE, Mageia and Fedora. We could automatically build some packages on > multiple distributions and be kinda upstream. So, this setup will need to distribute things... would it need mirrors as well? or ? > Idea №4: Building custom images with user content > > Deploy (or build) a tool(s) to enable user-built install, appliance and > container images with their content (modulo restrictions similar to COPR) > on top of Fedora. Yeah, this has been attempted about 3 times, but if someone has cycles to work on a app/tool again, great! > Implementation details > > - > > Koji and Koschei deployed in fedorainfracloud.org Nope. There are options of course, but the cloud won't be one. ;) We could do a remote cloud provider possibly, or we are considering another openshift cluster with kubevirt or perhaps just some simple virthost/ansible controlled thing. In any case I think if it's important we can get resources. > - > > A few builders constantly running, with a possibility to add more > builders as needed and as available cloud resources allow koji doesn't have much way to dynamically deal with builders... > - > > Deployed and maintained by proposal owners - not supported by Fedora > infrastructure > - > > Other contributors can have access granted by proposal owners (for the > time being only users in “packagers” group will be eligible to get access) > - > > Possibility to have some builders running outsides of Fedora > infrastructure - contributed by proposal owners I'd be carefull of this, as it can cause a lot of issues, but of course up to you as you are doing the work. :) > - > > Koji builds can be done from SCM (src.fp.o, pagure.io, github, …) or > from SRPM upload Ah, so it would not have it's own git/pagure? > > FAQWhy not COPR? > > - > > COPR has been starved of resources for years, which has impaired its > ability to provide reliable service at scale. Fedora Infrastructure refuses > to consider supporting it officially and Fedora Release Engineering seems > to have some undefined issues with COPR. Thats kind of glossing over a lot. The reason infrastructure doesn't want to support it is that its running on a ancient version of openstack thats very hard to fix when broken. The only issues I know otherwise is that copr is just designed differently from koji. koji is set to record everything and keep anything needed to rebuild things, but cope is much more free form. In any case with our cloud retirement there will be changes to copr. Not determined yet, but hopefully it will have more resources and a solid base to run on. For the case of this proposal, I agree copr might not be a good fit, as you are trying to fold all the changes back to koji, so best to just run a koji to make it as close as possible. > - > > It is not official build system of Fedora which is not helping with Problem > №2: Testing of new rpm/koji/mock features/configuration > <https://docs.google.com/document/d/1DtNTCa-gXc0ILDDHPyAcAca2GGGlENavfgPwGGgqJ_Y/edit#heading=h.iodoq4xw80c2> Note that if you also are testing new things you could break packagers that are trying to do other work... ...snip... > How can you make Koji builds faster? > > - > > By tuning Koji for performance, not correctness and reliability. > - > > By building only on a single, fast architecture. This may well bite you when you merge back to koji and build for all arches. ...snip... Overall sounds very nice... we will have to work out details if everyone is on board with it. Do note that it's going to be a lot of work for you all... :) kevin
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
