Hey Michal, On Sat, Nov 30, 2019 at 5:08 PM clime <clime@xxxxxxxxxxxxxxxxx> wrote: > > Hey Igor, > > On Tue, 26 Nov 2019 at 17:00, Igor Gnatenko > <ignatenkobrain@xxxxxxxxxxxxxxxxx> wrote: > > > > Hello fellows, > > > > After last publication on LWN about Fedora Modularity mess, I think it > > is time to describe the idea I was proposing internally with few other > > folks (Adam Samalik, Brian Exelbierd) back in the RH times. > > > > Before I actually go deep, I'll try to answer main questions to myself > > (so that you can understand why I am proposing this particular thing). > > > > 1. Do we want to package multiple streams only for "leaf" software or > > any kind of it? > > I believe that we need both, and we do support both. However, it might > > not look as nice as it could: > > * Need to create multiple repos for different "streams" > > * Need to maintain epel7/epel8/f30/f31/master branches > > * Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is > > manual work > > * However, those are supposed to be (according to the guidelines) > > parallel-installable (and not be conflicting in any way) > > Doing git merge / git cherry-pick and maintaining many git remotes > > requires some advanced git knowledge and a time. And we can't actually > > have multiple versions of a package with same name (without "mangled" > > names) in a repo due to the way how our buildsystem works (and not > > only buildsystem, with some caveats). > > We do have some kind of a solution for multiple releases building from > > one branch (package.cfg), however this work has been never finished, > > thus there are many problems with this approach. > > > > 2. Do we want to support buildtime-only packages? > > I would rather generalize this category as "less-supported packages". > > I maintain 800+ Rust packages and very often I need to update them to > > an incompatible version. In Rawhide I just do it, update all dependent > > packages to use new version, and if I can't do that for some reason, > > create "compat" package. Obviously, all patches are sent to the > > upstream. > > Upstreams are removing features, I need to deal with Obsoletes but I > > simply can't continuously add new Obsoletes into the > > fedora-obsolete-packages. And what for if they are used only during > > build of other, more important, packages? Why do I have to spend time > > with upgradepaths? I definitely want some mechanism which will tell to > > user that "THIS PACKAGE IS NOT FULLY SUPPORTED." > > Obviously, for packages which are used in runtime need a proper > > support as we do today for all packages to share work (that's the > > place where I agree with Kevin Kofler.) > > > > 3. Can we have different lifecycles for the software in Fedora? > > Right now we have to keep all versions of software which was there at > > GA point. And from the updates repo. We never remove packages > > entirely. That said, if package was there at GA time, it will have to > > be "supported" until the end of that Fedora release. > > I don't think we can (should?) do much in this regard. If we make > > packages to build from "stream" branch, we can put an information to > > that branch that this package should be built for all distributions > > (Fedora, EPEL) until this date. Or even store this info somewhere else > > like PDC (yes, I know we want to kill it). fedpkg build will take this > > into account and submit proper builds. And we can design some API in > > infra which would tell until when some particular stream of a package > > is supported. Much like it is done with Fedora releases & GNOME > > Software. > > > > 4. Do we want to have some kind of "stream expansion" where software > > builds against all Pythons, Perls and whatnot? > > I think this should be conscious choice of a distribution, and in > > specific cases, maintainer. Using some examples from previous threads, > > why does bugzilla have to be built against 2 different versions of > > perl and users could choose? I think maintainer should choose one > > version of perl and let bugzilla use it. Being able to build > > combinations of software is definitely nice, but I don't think this > > should be standard practice. openSUSE does that with ruby and python > > (they build all modules automatically for all versions) and I like > > this. But having all packages built against all is just combinatoric > > explosion. Given how many updates have feedback in Bodhi, I'm pretty > > sure 99% of combination won't be tested (or even installed?) ever. > > > > 5. Are we still trying to be a Linux distribution or we are just > > letting people to do whatever they want in our infrastructure? > > This question bothers me from time to time and I don't have answer. > > For example, Modularity is very flexible and I very often find people > > saying that you can expand this and that, but in Fedora we limit its > > usefulness. Do we actually need to develop something like this > > (knowing in advance that probably nobody outside of Fedora/RHEL will > > be using this software / technology)? Are we trying to create > > technologies which would be very extensible and used in other > > distributions instead of solving some specific problems? If former, > > why don't we talk to others about things in advance and not getting > > other people to work on these cool things? > > > > === > > > > How I would imagine having multiple streams: > > * rpms/nodejs has multiple branches - 10, 11, 12 > > * in all of them, nodejs.spec exists as-is without "mangled" name > > (just with different versions and such) > > * each has package.cfg or some alternative which specifies what is the > > EOL and/or for which releases to build > > * when builds are submitted, Koji automatically adds suffix to a main > > package and subpackages containing branch name > > * during the rpmbuild, extra provides are added (for the unversioned > > names and indication of a stream) and conflicts (possibly depending on > > some macro which user can disable) automatically > > > > Notes: > > * If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can > > describe this in standard way (Requires: (nodejs >= 9 with nodejs <= > > 11)) and one will be picked up automatically > > * If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10` > > * If that requires some conflicting dependency to be switched, it will > > be switched automatically > > * Packages produced from nodejs.src have Provides: stream(nodejs) > > stream(nodejs:9) and Conflicts: stream(nodejs) so that it is > > explicitly not possible to install 9 and 10 at the same time > > * If desired, packages can depend on a specific stream via Requires: > > stream(nodejs:10) without actually depending on nodejs (this requires > > some small piece of code in libsolv) > > * In the very similar way, user can lock themselves to a specific > > stream by writing some conf file which DNF would read (if point above > > is implemented, about just tens of lines of the code in libdnf, or > > even libsolv can be teached about that as well) > > * DNF can be teached about these special things so that you can > > automatically swap conflicting dependencies, but lock yourself to some > > streams of a package > > * All standard Conflicts/Requires/Obsoletes/… will simply work > > * With side tags on demand it is now much easier > > * Mass rebuild scripts will be teached about stream branches and will > > build packages properly > > * After branching, there will be a script which untags all packages > > which should not be supported in the new release > > * fedpkg will be teached about stream branches and will show which > > branches build where, so that maintainers can easily maintain them > > * Packaging guidelines should be adopted to accept conflicting > > packages and tooling should be improved to show the conflicts and how > > to resolve them > > * fedora-release will have Suggests: stream(nodejs:10) so that when > > user types dnf install nodejs, for example nodejs10 would be picked up > > (instead of nodejs12) > > * Some macros can be added to generate virtual package with "default > > stream", so that there will be actual nodejs package depending on > > nodejs12 if that would be default in given Fedora release > > * Going further, I would extend Koji with fedora plugin which would > > deal with the build submission instead of having this logic on clients > > so that all people get same result > > > > === > > > > Now let me quote Matthew Miller with the list of requirements and > > answer how we achieve them. > > > > 1. Users should have alternate streams of software available. > > I think this is achieved with what I have said above. We will ship > > nodejs9, nodejs10 at the same time. > > > > 2. Those alternate streams should be able to have different lifecycles. > > nodejs 9 and 10 can automatically build for all distros until EOL > > (which would be stored somewhere). EOL-date aware tooling (fedpkg) > > would not build for excluded versions of distros and build for distros > > where EOL of distro is further than EOL of component. > > > > 3. Packaging an individual stream for multiple outputs should be > > easier than before. > > As I wrote above, we will have one nodejs rpm repo with multiple > > branches (aka streams) and no name mangling needed. fedpkg build will > > spawn builds for all releases matching criteria (EOL date, excluded > > distributions). > > > > === > > > > This is not the exact approach we have tried back in the days, but it > > is very similar to that. Unfortunately I did not save any documents > > from those times but Adam or Brian probably still can find them (since > > they are RH internal). > > I agree that the approach is you described is good. From you have written, > it seems to me that Stream could be an explicit rpm property. Then the names > wouldn't be mangled just extended: ENSVRA. > > Anyway, i would just go with your approach for start. That way, one can avoid > bothering rpm team unless necessary. > > This is really the only sensible implementation of modularity (i.e. providing > multiple version of software) and the reason is that it provides the > functionality > required while leveraging already existing binary format to provide the > new content: RPM, which makes matters on user-side much easier. > > It would be good to make this change as it will send Fedora > (and all the other distros that use modularity) in the right direction again. > Persistently going the wrong way and pretending it is the right one > is not smart. +1 > clime > > > _______________________________________________ > > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
