As a meta-goal, we should break up "Modularity" into a number of separate components, some build-side, some client-side. Modularity suffers greatly from trying to encode everything into one document. This greatly raises the complexity of the task and makes it hard to consider alternative proposals for various parts. Let's consider them separately: 1. what to build against what branches You said: > * each has package.cfg or some alternative which specifies what is the > EOL and/or for which releases to build Right now we use dist-git branches for this. Having a package.cfg might sound easier , but the caveat is that building against different branches requires tweaks sometimes. So we immediately hit a choice: do we want a single branch and the %ifdef mess, or multiple branches and the cherry picking troubles. Right now, big disadvantage of multiple branches is that %changelogs and Release bumps introduce trivial conflicts. I think we should explore the proposals (incl. your and Neal's) to make this better. Fixing this would be good also for normal packaging, because it'd remove one or two more steps that need to be done every time. We should explore both directions, i.e. package.conf and easier branches. We should experiment with better/easier control of what to build where. This would benefit normal packaging too. Right now our tooling is just hard to use, with multiple commands in different tools needed to start builds, schedule updates, control koji status, etc. We need changes for "normal" packages as much as for "streams". 2. how to deliver rpms to users and how to let them control installed versions > How I would imagine having multiple streams: > * rpms/nodejs has multiple branches - 10, 11, 12 > * in all of them, nodejs.spec exists as-is without "mangled" name > (just with different versions and such) > * when builds are submitted, Koji automatically adds suffix to a main > package and subpackages containing branch name > * during the rpmbuild, extra provides are added (for the unversioned > names and indication of a stream) and conflicts (possibly depending on > some macro which user can disable) automatically Yes. The crucial part is that the rpms that are delivered are just normal rpms. All the complications are kept on the build-side and the client side doesn't need to care if this magic happened, the same result could have been achieved with manually mangled spec file. > * If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can > describe this in standard way (Requires: (nodejs >= 9 with nodejs <= > 11)) and one will be picked up automatically > ... > * All standard Conflicts/Requires/Obsoletes/… will simply work Yes. More of "just normal packages". > * If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10` > * If that requires some conflicting dependency to be switched, it will > be switched automatically > * Packages produced from nodejs.src have Provides: stream(nodejs) > stream(nodejs:9) and Conflicts: stream(nodejs) so that it is > explicitly not possible to install 9 and 10 at the same time > * If desired, packages can depend on a specific stream via Requires: > stream(nodejs:10) without actually depending on nodejs (this requires > some small piece of code in libsolv) > * In the very similar way, user can lock themselves to a specific > stream by writing some conf file which DNF would read (if point above > is implemented, about just tens of lines of the code in libdnf, or > even libsolv can be teached about that as well) > * DNF can be teached about these special things so that you can > automatically swap conflicting dependencies, but lock yourself to some > streams of a package ... and here I'm not sure. I'm not convinced we need to try to manage all this like that. We already have packages which Conflict, or specify conflicting Requires, and the user gives a dnf command, and a certain transaction is prepared for them. The transaction will occasionally specify a list of rpms to remove. If the user doesn't like this, they can cancel and give a different command. Specifically, some streams might conflict, other might not, etc. We could have a macro that helps to declare this in one line, but let the maintainers of the rpms declare this as needed. 3. [your 2.] Do we want to support buildtime-only packages? > I would rather generalize this category as "less-supported packages". > I maintain 800+ Rust packages and very often I need to update them to > an incompatible version. In Rawhide I just do it, update all dependent > packages to use new version, and if I can't do that for some reason, > create "compat" package. Obviously, all patches are sent to the > upstream. >[...] > * After branching, there will be a script which untags all packages > which should not be supported in the new release > > Upstreams are removing features, I need to deal with Obsoletes but I > simply can't continuously add new Obsoletes into the > fedora-obsolete-packages. And what for if they are used only during > build of other, more important, packages? Why do I have to spend time > with upgradepaths? I definitely want some mechanism which will tell to > user that "THIS PACKAGE IS NOT FULLY SUPPORTED." > Obviously, for packages which are used in runtime need a proper > support as we do today for all packages to share work (that's the > place where I agree with Kevin Kofler.) A different proposal: we add a new rpm flag (maybe as a special Requires: line) that specifies that a package is only supported as a build-time dependency. DNF could then gain a trivial patch that it would ask the user "Do you want to install those unsupported packages? If you do, please do not file bugs, because ...". The advantages: a) packages are available for local builds and downstream rebuilds b) no problem with mock and koji being different c) we avoid any perception of "welding down the engine hood". This would just acknowledge the truth that there are many packages in Fedora that are not supported beyond an occasional version bump and rebuild, giving better transparency for the users in general. 4. > 3. Can we have different lifecycles for the software in Fedora? a.k.a. Should we allow more in-release upgrades? > Right now we have to keep all versions of software which was there at > GA point. And from the updates repo. We never remove packages > entirely. That said, if package was there at GA time, it will have to > be "supported" until the end of that Fedora release. > I don't think we can (should?) do much in this regard. If we make > packages to build from "stream" branch, we can put an information to > that branch that this package should be built for all distributions > (Fedora, EPEL) until this date. Or even store this info somewhere else > like PDC (yes, I know we want to kill it). fedpkg build will take this > into account and submit proper builds. And we can design some API in > infra which would tell until when some particular stream of a package > is supported. Much like it is done with Fedora releases & GNOME > Software. We can (and do) encode this information directly in dist-git. No need to provide a meta-mechanism. Even if package.conf becomes a thing, this would contain this information. I think we should allow "rolling" versions more widely, like we do with firefox, the kernel, spam blockers, and probably some other things. As long some piece software is most backwards compatible, I think we'd be better of abandoning the strict guidelines we have now. This depends on each specific package though. If this happens, it should be as a FPC guideline change, discussed and specified independently of the tooling changes. If we detach it from the technical side, answer the questions when and if at all we allow this will be clearer. 5. > 4. Do we want to have some kind of "stream expansion" where software > builds against all Pythons, Perls and whatnot? > > I think this should be conscious choice of a distribution, and in > specific cases, maintainer. Using some examples from previous threads, > why does bugzilla have to be built against 2 different versions of > perl and users could choose? I think maintainer should choose one > version of perl and let bugzilla use it. Being able to build > combinations of software is definitely nice, but I don't think this > should be standard practice. openSUSE does that with ruby and python > (they build all modules automatically for all versions) and I like > this. But having all packages built against all is just combinatoric > explosion. Given how many updates have feedback in Bodhi, I'm pretty > sure 99% of combination won't be tested (or even installed?) ever. Right now this would be prohibitively expensive. But maybe this could become much easier if points 1-2 and 4 were better answered. So I think we should shelve this question for now. 6. How to deal with packages which need outdated BuildRequires This is another part of Modularity, and you proposal glosses over this. But you said: > * With side tags on demand it is now much easier ... and I fully agree here. There was https://fedoraproject.org/wiki/Changes/OnDemandSideTags which plays with this idea. I think side-tags could/should be the basis of a solution. > 5. Are we still trying to be a Linux distribution or we are just > letting people to do whatever they want in our infrastructure? > This question bothers me from time to time and I don't have answer. > For example, Modularity is very flexible and I very often find people > saying that you can expand this and that, but in Fedora we limit its > usefulness. Do we actually need to develop something like this > (knowing in advance that probably nobody outside of Fedora/RHEL will > be using this software / technology)? Are we trying to create > technologies which would be very extensible and used in other > distributions instead of solving some specific problems? If former, > why don't we talk to others about things in advance and not getting > other people to work on these cool things? I don't have an answer for this. > * fedpkg will be teached about stream branches and will show which > branches build where, so that maintainers can easily maintain them > * Packaging guidelines should be adopted to accept conflicting > packages and tooling should be improved to show the conflicts and how > to resolve them > * fedora-release will have Suggests: stream(nodejs:10) so that when > user types dnf install nodejs, for example nodejs10 would be picked up > (instead of nodejs12) > * Some macros can be added to generate virtual package with "default > stream", so that there will be actual nodejs package depending on > nodejs12 if that would be default in given Fedora release > * Going further, I would extend Koji with fedora plugin which would > deal with the build submission instead of having this logic on clients > so that all people get same result > > === > > Now let me quote Matthew Miller with the list of requirements and > answer how we achieve them. > > 1. Users should have alternate streams of software available. > I think this is achieved with what I have said above. We will ship > nodejs9, nodejs10 at the same time. > > 2. Those alternate streams should be able to have different lifecycles. > nodejs 9 and 10 can automatically build for all distros until EOL > (which would be stored somewhere). EOL-date aware tooling (fedpkg) > would not build for excluded versions of distros and build for distros > where EOL of distro is further than EOL of component. > > 3. Packaging an individual stream for multiple outputs should be > easier than before. > As I wrote above, we will have one nodejs rpm repo with multiple > branches (aka streams) and no name mangling needed. fedpkg build will > spawn builds for all releases matching criteria (EOL date, excluded > distributions). I think all this sounds reasonable. The main part is that it does not require massive changes in the koji or dnf, and that it integrates in a backwards-compatible way with what we have now. Let's split the problem into chunks and discuss solutions. > This is not the exact approach we have tried back in the days, but it > is very similar to that. Unfortunately I did not save any documents > from those times but Adam or Brian probably still can find them (since > they are RH internal). Zbyszek _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
