On Wed, Oct 16, 2019 at 9:00 PM Stephen Gallagher <sgallagh@xxxxxxxxxx> wrote: > > On Wed, Oct 16, 2019 at 8:44 PM Neal Gompa <ngompa13@xxxxxxxxx> wrote: > > > > On Wed, Oct 16, 2019 at 8:27 PM Stephen Gallagher <sgallagh@xxxxxxxxxx> wrote: > > > > > > On Wed, Oct 16, 2019 at 7:58 PM Kevin Kofler <kevin.kofler@xxxxxxxxx> wrote: > > > > > > > > Stephen Gallagher wrote: > > > > > An awful lot of people are repeating this as if it's a solution > > > > > without understanding the existing architecture. Believe it or not, > > > > > attempting to abandon default streams and go back to only non-modular > > > > > content available by default is a lot harder than it sounds (or should > > > > > be, but I noted that we're working on that in another reply elsewhere > > > > > in the thread). There is currently no path to upgrades that would get > > > > > back from the modular versions and the closest we could manage would > > > > > be to rely on the dist-upgrade distro-sync, but in that case we > > > > > *still* need to have DNF recognize that the default stream has changed > > > > > (in this case, been dropped) and handle that accordingly. > > > > > > > > So completely disable all module support in DNF by default with some global > > > > flag (make all the module code conditional under some new enable_modules > > > > flag and default the flag to enable_modules = 0), then it will treat the > > > > packages as normal packages and you only have to provide a higher EVR. All > > > > this module processing should only happen if the user explicitly enables it. > > > > > > > > > > Given that many of the modules in the distribution currently are there > > > specifically to provide newer, less-stable versions of the versions in > > > the non-modular/default stream, this would be fairly disastrous. For > > > example, Node.js 10.x is the default stream in Fedora 30 because it's > > > the LTS branch. It also provides a non-default stream for 8.x (the > > > previous LTS branch that a lot of applications still rely on) and > > > 12.x, which will be the next LTS branch in November, but is not > > > guaranteed stable yet. With the approach you're describing, everyone > > > would be forcibly updated to the unstable 12.x release. > > > > > > The only way we could assure ourselves that this wouldn't happen would > > > be to do another mass-rebuild, bumping the epoch of every package that > > > exists in a module. That's a lot of work. > > > > > > > > > > We could let "dnf distro-sync" take care of it. Rebuilds to remove > > RPMTAG_MODULARITYLABEL from the package headers would be necessary, > > but otherwise nothing else should need to change. > > > > That would still lead to upgrading to the highest NEVRA though. Which > is problematic as I mentioned above. > It'd be interesting if an "inverse filter" could be applied. Instead of modules shadowing non-modular content, the other way around would occur. That would make it easy to clean that up. > > > > A slightly more elaborate, but slightly harder to implement, approach would > > > > be to let DNF simply disable modules that are enabled locally but no longer > > > > available in the repositories, together with disabling the fedora-modular > > > > and updates-modular repositories by default. > > > > > > > > > > And again, this only works if every packager who has spent time > > > creating a module with a default stream takes their content and shoves > > > it back into the non-modular repository. Which in some cases they > > > probably cannot do, because they have build-dependencies that are in > > > conflict. This is a highly non-trivial process and it would need to be > > > done individually for every single package. That's far more > > > packager-hostile than fixing the default stream/buildroot problem and > > > the upgrade path problem. > > > > > > > And the case of demodularizing packages has to be addressed sooner or later > > > > anyway, so better address it sooner rather than later, before more and more > > > > damage is done by maintainers moving packages to module-only without a way > > > > back. > > > > > > > > > > I've already acknowledged upthread that demodularizing packages is a > > > problem we need to solve. It's being worked on, but it's a lot harder > > > than you think, because we have failsafe code implemented in libdnf to > > > prevent *accidental* demodularization that's in conflict with this > > > desire. Also, this paragraph was needlessly antagonistic: moving > > > packages to modules is not "damage". > > > > > > > > > > It was damaging when it was happening before we have a way to depend > > on modules from non-modular content. It essentially forces other > > packagers to move to modules too. It's a snowball effect. And *right > > now* modularization is a one way road. I'm pleased to hear that we > > will get a way to demodularize, but currently we don't have it. > > > > That's a fair observation. I can only plead my team's lack of > omnipotence and our willingness to correct our mistakes. > Sure. And I know personally that you guys were far too compressed to figure out this problem. It would have not shown up in your thought processes, given the focus for the past 18 months. > > > > > I started this discussion to ask the community to help us identify the > > > > > best path *forward*. An endless barrage of "kill it off" replies is > > > > > not helpful or productive. If anyone has specific advice on how to > > > > > move forward (or, indeed, if you can figure out how to migrate back > > > > > without considerable release engineering and packager effort), that > > > > > would be productive. Just please keep in mind that we have to go to > > > > > war with the army we have, not the one we wish we had. > > > > > > > > If you are standing in front of a cliff, moving forward is just not the > > > > answer. Not all changes are improvements. Sometimes, you have to realize > > > > that you made a mistake and move back before things only get worse. > > > > > > > > > > Sure, but we are nowhere near a cliff. As I just posted in the Change > > > Proposal thread, there are three problems we need to solve, two of > > > which we already have solutions designed for and one (this thread) > > > that we are trying to finalize. That's far from "standing in front of > > > a cliff". > > > > > > Please understand that "I don't understand how this works" is not the > > > same thing as "This doesn't work". > > > > > > > The overwhelmingly negative feedback that you are getting is a clear > > > > indication that something is wrong. You should not ignore it or summarily > > > > file it off as luddites wanting to return to the past. There are real issues > > > > with modules, and the Modularity WG is only offering partial workarounds > > > > (adding more and more complexity) and no real fixes. > > > > > > > > > > So, literally every word of this is wrong. The negative feedback is > > > not "overwhelming". It is approximately four noisy individuals, all of > > > whom have expressed zero interest in understanding the actual > > > situation that they are trying to "fix" by endlessly insulting the > > > people working on the problem. Demoralizing the people who can dig us > > > out of this situation is an unwise strategy. > > > > > > Also, we're not offering "partial workarounds" (excepting some > > > acknowledged hackery to avoid blocking F31). All of the proposals I > > > have been discussing in this thread are for real design adjustments > > > for long-term benefit. And while they add some additional complexity > > > on the *infrastructure*, a primary goal is to not make the users or > > > packagers' lives harder. We *know* that the default stream/buildroot > > > issue is failing to hit this goal and the solution is known, > > > implemented upstream and could be deployed by the end of the week if > > > FESCo gives its approval. > > > > > > > I think we must have a good UX for handling module transitions. And I > > know you've mentioned upthread that you want this to be a painful > > experience, > > To be clear, I want it to be a painful experience for *arbitrary* > module transitions. I think it absolutely needs to have good UX for > *planned* and *tested* transitions. I have proposed some ideas for > that. > > > but I would argue that it should not be. We have gone to > > great lengths to smoothly handle transitions for non-modular content > > for the past five years. I think we should consider that modules > > should grow the same facilities. The moment you made modules have > > dependencies, you basically set it up for requiring the full > > dependency expression model that RPM has. > > > > At minimum, we need the following: > > * Provides > > * Requires > > * Obsoletes > > > > Without these three, we can't do modular transitions cleanly across releases. > > > > "Begging the question": You're asserting this without context. I > proposed something upthread (as a direct reply to my original message) > involving "upgrades:" and "obsoletes:", because I think that might be > a cleaner approach than just relying on the default stream of the > repos you have enabled. I don't know that Provides and Requires are > useful though. Explain it to me, please? > Provides is needed because once modules are replaced, the module dependency expressions for third party modules should be satisfiable if they are API-compatible (in the normal sense). Ripping modules out of a distro is going to break things from an ISV perspective unless a mechanism is in place for something else to "slot in", so to speak. This is obviously paired with "Obsoletes", which I think is self-explanatory by now. Another case for "Provides" would be alternative implementations of the same module. For example, a module of Java 11 could be done with Oracle Java (blech), OpenJDK, or OpenJ9. In all three variants, they are fully substitutable because they provide the same interfaces that other modules can depend on. But they aren't the same modules, because they're built on different "cores". Requires is kind of obvious, we sort of already have it, it's just not well-defined. > > > We also need the platform module runtime dependency to become an > > optional property. In a "modular" world, it's going to be impossible > > to get rid of modules to upgrade across system releases, so we need > > modules to not be tightly bound to the platform when they don't need > > to be. The underlying property here should be split in two, > > effectively BuildRequires and Requires. > > > > It needs to be possible to have orphaned modules on the system. > > Without that, smooth and seamless system upgrades are going to be > > *very* hard. We've never done this to non-modular packages, it's kind > > of insane to do that for modular ones. > > > > That's an interesting thought and one I hadn't seen put to words yet. > It is certainly worth exploring what cases that would benefit. Do you > have some examples you could share? > In Rust, we have a lot of leaf modules. They ultimately provide an application that depends on stable interfaces that are binary compatible as you move forward (glibc, etc.). From that perspective, we should be able to treat modular software the same way we treat non-modular software. And realistically speaking, modules should be able to stick around unless they can't. There's some complexity here because of the whole parallel-availability bit, but I think it's critical for sustainably maintainable platform for us to be able to do this. > Currently, our default stance has been "disallow the system upgrade if > the modules they've locked onto won't be available there". This is > based on our philosophy that ultimately "the app is what matters". > Most people don't install Linux because they enjoy clicking buttons in > Anaconda. They install Linux because they have an application they > want to deploy. We want our upgrade process to be focused on *keeping > that app running*. When possible, we want them to be able to say "I > need Node.js 10.x" and even if the system default becomes 12.x or > 14.x, as long as a 10.x stream exists in the next Fedora release, they > should be able to upgrade their base system without breaking their > application. With that philosophy in mind, blocking the upgrade seems > more user-friendly than allowing the upgrade to proceed with a > possibly-unusable Node.js 10.x installation on the system. I'd rather > they see that they have a conflict to resolve and deal with either > porting their system to the newer Node.js stream or else switch to a > distro like RHEL which will maintain that stream past its upstream > EOL. > It is incredibly rare that a situation you've contrived (though a valid one!) actually happens. In nearly all cases that I've been doing this, packages built for older Fedora releases don't immediately "go bad" when you upgrade. If they have no upgrade candidate and their dependencies are no longer satisfiable, then there's a problem. Otherwise, it's just more "orphaned" stuff that you can continue to use unless it's being replaced or upgraded. If a nodejs 10 stream exists in the F+1 release, then keep it. But if it's going away and the maintainer has set it up to move to nodejs 12, then that should happen automatically, *regardless* of whether it's default or not, unless you want to permit orphaned modules, then it could be a switch that triggers the upgrade of modules vs orphaning them. If the nodejs 10 stream goes away without an upgrade path in F+1, then it should be able to remain on the system if the RPM level dependencies are still satisfied in F+1, which goes toward that "app primacy" philosophy. Blocking upgrades is dangerous, because users usually don't have enough information to solve the problem in the way you're hoping. > > -- > > 真実はいつも一つ！/ Always, there's only one truth! > > But an infinite number of observer biases! Heh, indeed. That's the first time in years someone has commented on my email signature. :) -- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
