Hi Neal, On Sat, Nov 30, 2019 at 11:58 PM Neal Gompa <ngompa13@xxxxxxxxx> wrote: > > On Tue, Nov 26, 2019 at 10:59 AM Igor Gnatenko > <ignatenkobrain@xxxxxxxxxxxxxxxxx> wrote: > > > > Hello fellows, > > > > After last publication on LWN about Fedora Modularity mess, I think it > > is time to describe the idea I was proposing internally with few other > > folks (Adam Samalik, Brian Exelbierd) back in the RH times. > > > > Before I actually go deep, I'll try to answer main questions to myself > > (so that you can understand why I am proposing this particular thing). > > > > 1. Do we want to package multiple streams only for "leaf" software or > > any kind of it? > > I believe that we need both, and we do support both. However, it might > > not look as nice as it could: > > * Need to create multiple repos for different "streams" > > * Need to maintain epel7/epel8/f30/f31/master branches > > * Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is > > manual work > > * However, those are supposed to be (according to the guidelines) > > parallel-installable (and not be conflicting in any way) > > Doing git merge / git cherry-pick and maintaining many git remotes > > requires some advanced git knowledge and a time. And we can't actually > > have multiple versions of a package with same name (without "mangled" > > names) in a repo due to the way how our buildsystem works (and not > > only buildsystem, with some caveats). > > We do have some kind of a solution for multiple releases building from > > one branch (package.cfg), however this work has been never finished, > > thus there are many problems with this approach. > > > > I think we need to recognize that we've done some poor optimization > for the majority of packager workflows. Even if we consider modules, > the vast majority of components will never be modularized. Moreover, > we already know that the overwhelming majority of specs are managed > identically across branches. Yeah, indeed. I think this (optimization of packager workflows) was never an explicit goal of people working on Modularity. > In these cases, we can make fedpkg better for handling this. There's > no reason that people can't have a single master branch and only > branch for distro releases as needed. > > I'm trying this workflow myself with libeconf[1], but unfortunately > fedpkg doesn't handle this model well right now. I proposed a > particular feature request for this[2], but there might be other ways > to make this work. The current approach for package.cfg is terribly > broken, though. It definitely needs more thinking and probably requires more concrete proposal how it should work overall. Not just "if package.cfg is there, parse it and build for targets specified in there". I'm also not entirely sure if we should encode this information in the git repo as a file because it is not possible to query this information (e.g. to find out which branches build where) and do checks (e.g. that you don't build 1.x and 2.x in the f31 and you choose not to mangle names). > As for the name mangling, I think this is a relatively minor issue for > supporting multiple versions of software. There *are* things we could > do to minimize this, but it's not a good point to focus on right now. > > [1]: https://src.fedoraproject.org/rpms/libeconf > [2]: https://pagure.io/fedpkg/issue/352 > > > 2. Do we want to support buildtime-only packages? > > I would rather generalize this category as "less-supported packages". > > I maintain 800+ Rust packages and very often I need to update them to > > an incompatible version. In Rawhide I just do it, update all dependent > > packages to use new version, and if I can't do that for some reason, > > create "compat" package. Obviously, all patches are sent to the > > upstream. > > I think we should categorically disallow > build-time-only/buildroot-only packages. They don't make sense in a > Fedora context and make it impossible to share resources. What's worse > with this model as it is currently implemented is that nobody can > figure out whether there are buildroot-only packages involved, and > multiple maintainers of packages basically can't happen. Well, if we make easier builds for multiple targets (without pushing 5 times, creating 10 overrides and waiting for repos) and relax update policy for such packages, then I definitely don't mind having them in the main repo. > Re-introducing those buildroot-only packages back into the main > package collection gives people an opportunity to depend on them > freely and co-maintain them. Now, we can't force anyone to co-maintain > packages, but as we see with the large number of orphanings that have > been happening lately, if people actually want or need packages, they > tend to take them on too. That's because those packages are required in the runtime, while I was talking about case like with Rust packages which are used only in build time and linked statically. > I think we can make it easier to support shipping these things to > stable releases too. The ability to use side tags to map to Bodhi > update submissions is going to be a good step forward there. Sadly, Bodhi maintainers so far said that they don't plan to enable submitting updates from side tags in non-rawhide :/ > > Upstreams are removing features, I need to deal with Obsoletes but I > > simply can't continuously add new Obsoletes into the > > fedora-obsolete-packages. And what for if they are used only during > > build of other, more important, packages? Why do I have to spend time > > with upgradepaths? I definitely want some mechanism which will tell to > > user that "THIS PACKAGE IS NOT FULLY SUPPORTED." > > Obviously, for packages which are used in runtime need a proper > > support as we do today for all packages to share work (that's the > > place where I agree with Kevin Kofler.) > > > > I'm not sure I want to have a concept of "not fully supported" (do we > have a concept of supported, really?). But at least for the RHEL > context, it makes sense to consider it. An RPM tag for setting that > data could be added. I think it probably makes sense to have an RPM > tag where we could declare supportability. Something less dumb than > the RPMTAG_MODULARITYLABEL tag that would be more generically > useful... We do have some update policy which would prevent me from updating packages and I simply could not maintain old versions and fix CVEs. Neither it makes any sense since I will need new version anyway. > Short term, we could "cheat" by adding Provides, but I think a tag > would be better since we can easily pass a definition to rpmbuild for > a package build. Adding tags to RPM is relatively easy, but we should > convince the RPM developers to do it. > > As for the whole Obsoletes thing, I think you could have some kind of > dynamically generated package that is set up to self-destruct and > include those Obsoletes. Templating a variation of the > fedora-obsolete-packages package that could be updated and generate > all the needed Obsoletes should be possible. Ordinarily, these > Obsoletes would be part of the package it replaces, but at the Rust > ecosystem scale, it makes sense to have a self-destructing package > that does that centrally. How we'd specifically implement this? I > don't know. Well, if you have Provides: unsupported(), then you can hook DNF to set SOLVER_ALLOWUNINSTALL + SOLVER_FORCEBEST for such packages, so they won't stay on older version and if they conflict, they would be removed without user intervention of --best --allowerasing. What do you think? That's probably what we should have done with python2 packages instead of adding thousands of lines into the fedora-obsolete-packages :) > > 3. Can we have different lifecycles for the software in Fedora? > > Right now we have to keep all versions of software which was there at > > GA point. And from the updates repo. We never remove packages > > entirely. That said, if package was there at GA time, it will have to > > be "supported" until the end of that Fedora release. > > I don't think we can (should?) do much in this regard. If we make > > packages to build from "stream" branch, we can put an information to > > that branch that this package should be built for all distributions > > (Fedora, EPEL) until this date. Or even store this info somewhere else > > like PDC (yes, I know we want to kill it). fedpkg build will take this > > into account and submit proper builds. And we can design some API in > > infra which would tell until when some particular stream of a package > > is supported. Much like it is done with Fedora releases & GNOME > > Software. > > > > Software shipped in *any* default enabled repositories that other > packages can depend on *must* be available for the life of a Fedora > release. > > I think we could be more flexible for software shipped in repositories > not enabled by default. But today, fedora-modular is enabled by > default and is now supported as a buildroot input. So that's > completely toast. > > For EPEL, we need to have RHEL-like flexibility for supportability. > Ideally, we could compose the supportability ranges for components > (i.e. packages) through the RPM tag or Provides... Can we actually create multiple repos and ship them? I don't think our tooling is ready for this. That would be quite some manual work (pungi, mirrormanager?). > Regardless, if we consider the existing modularity system, we can't > actually set SLAs right now anyway, so this is a problem we need to > solve for any solution. > > > 4. Do we want to have some kind of "stream expansion" where software > > builds against all Pythons, Perls and whatnot? > > I think this should be conscious choice of a distribution, and in > > specific cases, maintainer. Using some examples from previous threads, > > why does bugzilla have to be built against 2 different versions of > > perl and users could choose? I think maintainer should choose one > > version of perl and let bugzilla use it. Being able to build > > combinations of software is definitely nice, but I don't think this > > should be standard practice. openSUSE does that with ruby and python > > (they build all modules automatically for all versions) and I like > > this. But having all packages built against all is just combinatoric > > explosion. Given how many updates have feedback in Bodhi, I'm pretty > > sure 99% of combination won't be tested (or even installed?) ever. > > > > I think we should be able to do this, but Fedora probably doesn't have > near the build system resources that openSUSE has to pull it off. > > To give an idea of the disparity here, there's somewhere around ~100 > build machines that can take on ~8-10 builds at a time for x86 builds. > There's 7 POWER machines that can take on ~35 builds each for ppc64le. > There's 30 s390x LPARs that can run ~5-10 builds each. There's ~30 ARM > machines that can take on ~5-10 builds at a time for armv7hl/aarch64 > builds. > > We compete favorably for POWER and match up for s390x and ARM, but we > are massively outclassed on x86 builds. Even if you consider that the > large chunk of noarch packages we have to build can largely be built > on any architecture, packagers get penalized for it building on ARM > machines (because they're awful, unfortunately, and they're *never* > going to get better because the ARM ecosystem doesn't really care > about performance in the same way that all the other supported > architectures do). Even if we were forcibly scheduling noarch builds > on x86, we'd still wipe out our resources easily with stream > expansion. > > We already have problems with existing module builds wiping out all > the available build resources by pushing hundreds of builds without > any reasonable scheduling, ordering, or prioritization. We need to > deal with the resourcing and scheduling issues before we can consider > doing this on a broad scale. So far it did not kill Koji, right? :) Though the question was whether we *want*, not whether we have enough resources to do so. > > 5. Are we still trying to be a Linux distribution or we are just > > letting people to do whatever they want in our infrastructure? > > This question bothers me from time to time and I don't have answer. > > For example, Modularity is very flexible and I very often find people > > saying that you can expand this and that, but in Fedora we limit its > > usefulness. Do we actually need to develop something like this > > (knowing in advance that probably nobody outside of Fedora/RHEL will > > be using this software / technology)? Are we trying to create > > technologies which would be very extensible and used in other > > distributions instead of solving some specific problems? If former, > > why don't we talk to others about things in advance and not getting > > other people to work on these cool things? > > > > The ideal situation would be that we'd develop technology that all > distributions would like to use. > > But, of course, that requires something that the RPM distribution > communities aren't used to doing: collaborating at the early stage. We > did it with Rust, but that was an unusual event. It doesn't help that > the two major RPM distributions are commercial rivals and the two > companies aren't used to collaborating with each other on these kinds > of things. Correct me, but I don't think that build infrastructure is the place where Red Hat and SUSE are "fighting". Kernel, live patching, container tools, virt tools, long-term support and things like that makes money. So I don't think anybody needs to waste people's time on such things. > Fortunately, there *are* some links among the distributions on the > community side. But the modularity project started from within Red > Hat, and I suspect there wasn't any talking to SUSE in the early > stages of the development. There wasn't much community solicitation to > Fedora in the beginning either, as we know... Yeah, and that is pretty sad. > I hope that we can recover from this and develop a solution that all > distributions would like to use. +1 > > === > > > > How I would imagine having multiple streams: > > * rpms/nodejs has multiple branches - 10, 11, 12 > > * in all of them, nodejs.spec exists as-is without "mangled" name > > (just with different versions and such) > > * each has package.cfg or some alternative which specifies what is the > > EOL and/or for which releases to build > > * when builds are submitted, Koji automatically adds suffix to a main > > package and subpackages containing branch name > > * during the rpmbuild, extra provides are added (for the unversioned > > names and indication of a stream) and conflicts (possibly depending on > > some macro which user can disable) automatically > > > > We could do this with dynamically configured dep generators injected > in to the buildroot or such along with other tweaks throughout the > build process. Yeah, I actually have done that :) > > Notes: > > * If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can > > describe this in standard way (Requires: (nodejs >= 9 with nodejs <= > > 11)) and one will be picked up automatically > > * If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10` > > * If that requires some conflicting dependency to be switched, it will > > be switched automatically > > * Packages produced from nodejs.src have Provides: stream(nodejs) > > stream(nodejs:9) and Conflicts: stream(nodejs) so that it is > > explicitly not possible to install 9 and 10 at the same time > > * If desired, packages can depend on a specific stream via Requires: > > stream(nodejs:10) without actually depending on nodejs (this requires > > some small piece of code in libsolv) > > * In the very similar way, user can lock themselves to a specific > > stream by writing some conf file which DNF would read (if point above > > is implemented, about just tens of lines of the code in libdnf, or > > even libsolv can be teached about that as well) > > * DNF can be teached about these special things so that you can > > automatically swap conflicting dependencies, but lock yourself to some > > streams of a package > > * All standard Conflicts/Requires/Obsoletes/… will simply work > > * With side tags on demand it is now much easier > > * Mass rebuild scripts will be teached about stream branches and will > > build packages properly > > * After branching, there will be a script which untags all packages > > which should not be supported in the new release > > * fedpkg will be teached about stream branches and will show which > > branches build where, so that maintainers can easily maintain them > > * Packaging guidelines should be adopted to accept conflicting > > packages and tooling should be improved to show the conflicts and how > > to resolve them > > We already allow this, so I think we're fine. DNF needs some better > representation and advice for what to do in these scenarios, but > that's fixable. :) +1 > > * fedora-release will have Suggests: stream(nodejs:10) so that when > > user types dnf install nodejs, for example nodejs10 would be picked up > > (instead of nodejs12) > > I'm reluctant to stuff things into fedora-release. I think it'd work > equally well if we had a fedora-modular-release to define these. I've > used a variation of this technique in Mageia (mageia-repos-pkgprefs) > and OpenMandriva (openmandriva-repos-pkgprefs), so I know it would > work. Well, that's implementation detail. Just wanted to point out that we *can* do that. > > * Some macros can be added to generate virtual package with "default > > stream", so that there will be actual nodejs package depending on > > nodejs12 if that would be default in given Fedora release > > Makes sense to me. It also means it's easy to change things midway > through (which would be necessary in RHEL land) > > > * Going further, I would extend Koji with fedora plugin which would > > deal with the build submission instead of having this logic on clients > > so that all people get same result > > > > I'm not sure what you mean here... So instead of "fedpkg build" looking at package.cfg and deciding for which targets to build, it would just call to Koji and that would spawn multiple builds. The problem with former approach is that everyone has to have latest version of fedpkg and you can't force people to do so. > > === > > > > Now let me quote Matthew Miller with the list of requirements and > > answer how we achieve them. > > > > 1. Users should have alternate streams of software available. > > I think this is achieved with what I have said above. We will ship > > nodejs9, nodejs10 at the same time. > > > > 2. Those alternate streams should be able to have different lifecycles. > > nodejs 9 and 10 can automatically build for all distros until EOL > > (which would be stored somewhere). EOL-date aware tooling (fedpkg) > > would not build for excluded versions of distros and build for distros > > where EOL of distro is further than EOL of component. > > > > 3. Packaging an individual stream for multiple outputs should be > > easier than before. > > As I wrote above, we will have one nodejs rpm repo with multiple > > branches (aka streams) and no name mangling needed. fedpkg build will > > spawn builds for all releases matching criteria (EOL date, excluded > > distributions). > > > > === > > > > This is not the exact approach we have tried back in the days, but it > > is very similar to that. Unfortunately I did not save any documents > > from those times but Adam or Brian probably still can find them (since > > they are RH internal). > > I'm curious what the original proposal was, but these are very solid ideas... > > > > -- > 真実はいつも一つ！/ Always, there's only one truth! > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
