On Thu, 2019-10-17 at 09:47 -0400, Stephen Gallagher wrote:
> On Wed, Oct 16, 2019 at 9:39 PM Kevin Kofler <kevin.kofler@xxxxxxxxx> wrote:
> > So it looks like I did not describe clearly enough what my proposed
> > enable_modules=0 flag would do. ("Disable all module code" was apparently
> > too vague.)
> >
> > How I think it should work would be:
> > * For repositories, it completely ignores modular metadata and processes
> > only the non-modular parts of the repository metadata. Therefore, it does
> > not see the Node.js 12.x stream at all. It only sees whatever Node.js is
> > in the non-modular repository. If there are currently only modular
> > versions, then it sees none at all. But with the proposal to require a
> > non-modular default version, it would then see that version, which would
> > likely be Node.js 10.x. Nobody would get forcefully upgraded from 10.x to
> > 12.x.
> > * For installed modules, it completely ignores them, acting as if the
> > database of installed modules were empty. (It just does not read that
> > database at all.)
> > * For installed packages, it treats them all as non-modular. Sure, packages
> > originally installed from a module have weird EVRs encoding module
> > metadata, but otherwise they get processed exactly like a non-modular
> > package. So the default repository only has to provide a newer EVR to
> > upgrade the package. That should address upgrades from 8.x or 10.x to the
> > new default 10.x. If the user had previously installed 12.x, they will
> > only get downgraded if they distro-sync or if package-level dependency
> > issues with the F30 build of 12.x on F31 force the downgrade.
> >
> > I hope that clears it up.
>
> It does, thanks. I think you're right, that probably *would* work,
> though it's slightly harder to do your third bullet point than it
> sounds at first blush. I suppose we could fudge it with the
> `module_hotfix` option though..
>
> > > > A slightly more elaborate, but slightly harder to implement, approach
> > > > would be to let DNF simply disable modules that are enabled locally but
> > > > no longer available in the repositories, together with disabling the
> > > > fedora-modular and updates-modular repositories by default.
> > >
> > > And again, this only works if every packager who has spent time
> > > creating a module with a default stream takes their content and shoves
> > > it back into the non-modular repository. Which in some cases they
> > > probably cannot do, because they have build-dependencies that are in
> > > conflict. This is a highly non-trivial process and it would need to be
> > > done individually for every single package. That's far more
> > > packager-hostile than fixing the default stream/buildroot problem and
> > > the upgrade path problem.
> >
> > You are overestimating by far the effort required to demodularize the
> > handful packages that are currently module-only. The evidence Fabio
> > Valentini has gathered so far shows that actually very few packages would be
> > affected and they would not be too hard to fix. And Miro has also offered
> > help with fixing affected packages.
> >
> > All in all, it would require fixing a handful packages once and for all
> > instead of implementing workarounds affecting the entire distribution and
> > its thousands of users.
> >
>
> It's worth considering. I'm not ruling it out at this time. I'm not
> committed to doing it yet either.
>
> > > > And the case of demodularizing packages has to be addressed sooner or
> > > > later anyway, so better address it sooner rather than later, before more
> > > > and more damage is done by maintainers moving packages to module-only
> > > > without a way back.
> > >
> > > I've already acknowledged upthread that demodularizing packages is a
> > > problem we need to solve. It's being worked on, but it's a lot harder
> > > than you think, because we have failsafe code implemented in libdnf to
> > > prevent *accidental* demodularization that's in conflict with this
> > > desire.
> >
> > And exactly that code needs to go, at least in Fedora. I think having a way
> > to migrate away from modules is the common case to prioritize here.
> >
>
> This is one of those cases where I think that RHEL and Fedora needs
> are in conflict; in RHEL, we absolutely need to support the failsafe
> behavior, because accidentally replacing a critical dependency will
> break user applications. In Fedora, this is likely a smaller concern.
> It needs investigation.
>
>
> > That said, it shall be pointed out that, if the proposal to demodularize all
> > default versions of packages gets implemented, we only need a *short term*
> > solution for demodularization in DNF. After 2 releases, we have no default
> > streams left (and they will never come back by policy) and we can expect
> > users to have upgraded through a release with no default streams (given that
> > we do not support upgrading directly to n+3), so at that point DNF can
> > revert to the "safe" behavior (preventing accidental demodularization) by
> > default.
> >
>
> If we settle on the "no content in default streams" policy for Fedora,
> this is a sensible way to go about it, yes.
>
> > So the proposal to demodularize everything could actually make this problem
> > easier to solve.
> >
> > > Also, this paragraph was needlessly antagonistic: moving packages to
> > > modules is not "damage".
> >
> > Sorry for that. The reason I called it "damage" being done is that there is
> > currently no supported way back (as you pointed out yourself) and that it
> > moves us away from the state I (and others, e.g., Miro) believe we should
> > reach (where there are no module-only packages anymore).
> >
>
> Yeah, I can understand that perspective.
>
>
> > I consider this approach of making a controversial and experimental change
> > with no contingency plan, then using that absence of a contingency plan as
> > an argument to not only stick to the change at all costs, but even go
> > further with it, entirely unacceptable. (We call it the "creating facts"
> > ("Fakten schaffen") tactic in the German-speaking parts of the world. It is
> > an effective way to bypass discussion and democratic participation.)
> >
>
> Yeah, I'm not trying to make this a "fait accompli" discussion either.
> We are where we are. I just don't want to revert as a knee-jerk
> reaction if we can find a sustainable solution. I do understand your
> frustrations in the way things have happened thus far. I'm not
> entirely convinced that killing off default streams is the right
> approach, but you and Miro have made enough compelling arguments that
> it has to be considered carefully.
>
> > > Sure, but we are nowhere near a cliff. As I just posted in the Change
> > > Proposal thread, there are three problems we need to solve, two of
> > > which we already have solutions designed for and one (this thread)
> > > that we are trying to finalize. That's far from "standing in front of
> > > a cliff".
> >
> > The problems you listed are not the only ones. There are the version
> > conflicts between non-default streams that can be dragged in by default
> > streams.
>
> I submit that this is true of the non-modular content as well. We've
> worked around this in some places with the alternatives system, but
> there are plenty of cases where installing an RPM means that some
> other RPMs are now in conflict. This just moves it to another layer.
>
> > There is the buildroot-only content, which breaks our self-hosting
> > promise and makes it harder to compile additional software.
>
> We don't really have a self-hosting promise; We have an aspiration. We
> know that in reality, if we tried to do a mass-rebuild at Final
> Freeze, we would fail to build parts of the OS. We can only assert
> that *at some time* we were able to build everything with other
> packages available in Fedora. Also, the Ursa Prime Change will allow
> us to ship this in the buildroot repository publicly, so I think that
> is addressed.
>
> > There is the
> > issue that modules built for an older distribution cannot be kept on a newer
> > distribution version, which is a regression from ursine packages that (e.g.,
> > if they get dropped from the distribution) you can keep as long as they
> > don't depend on an outdated soname, which can be decades. There is the risk
> > that your more and more complex solutions introduce new issues, e.g., bugs
> > and undesirable behavior in DNF, that may or may not be easy to fix. (The
> > more complexity you introduce, the harder it gets for DNF to behave the way
> > the user expects.)
>
> Would you mind opening a ticket at https://pagure.io/modularity/issues
> and/or a new mail thread on this? I think there's value to what you're
> suggesting. We need to figure out what the right experience is and I
> think this thread is the wrong place for it.
>
> > And this needs to be brought into relation with the benefits of using
> > default streams instead of non-modular default versions, which as far as I
> > can tell are essentially nonexistent. (Sure, they might be an easier upgrade
> > path for things that are already default streams now, but this is self-
> > referential. We should not stick to default streams forever only because of
> > a historical decision. And actually, the upgrade path from default stream to
> > default stream is also not working yet, which is why we have this thread at
> > all.)
>
> As Alexander pointed out elsewhere in the thread, there *are* other
> benefits to being able to customize the buildroot. Some of them can be
> emulated with side-tags, buildroot overrides and compat packages, but
> that's pretty complex. If we make such a choice, we need to understand
> what we are losing.
>
> > > Please understand that "I don't understand how this works" is not the
> > > same thing as "This doesn't work".
> >
> > I understand very well how this doesn't work. :-)
> >
>
> Touché
>
> > > So, literally every word of this is wrong. The negative feedback is
> > > not "overwhelming". It is approximately four noisy individuals, all of
> > > whom have expressed zero interest in understanding the actual
> > > situation that they are trying to "fix" by endlessly insulting the
> > > people working on the problem. Demoralizing the people who can dig us
> > > out of this situation is an unwise strategy.
> >
> > LOL, filing off mailing list consensus as "four noisy individuals". This is
> > getting really ridiculous! Why am I even arguing with somebody who clearly
> > does not want to listen?
> >
>
> See my other reply. This was me getting frustrated while being
> overtired. Please accept my apology for that outburst.
>
> > And you are the people who brought us into this situation to begin with.
> > Default streams should never have been allowed without:
> > 1. an upgrade path from default stream to default stream, AND
>
> This was a lack of foresight. We didn't account for this and didn't
> realize it was a problem until users started complaining that they
> expected it. We can't think of everything.
>
> > 2. a contingency plan, i.e., an upgrade path from default stream to
> > non-modular default version
>
> Pretty much the same answer as above. It's easy with hindsight to say
> these were obvious and expected, but we are only human.
>
> > The fact that they were implemented without EITHER of these (when actually
> > BOTH are needed) was extremely short-sighted.
> >
> > So bringing yourselves up now as "the people who can dig us out of this
> > situation" feels to me feels like intentionally making a patient sick so you
> > can "cure" them.
> >
>
> I think that's a little harsh (but probably fair given my tone above).
> Can we agree that we're both on the same side: we want Fedora to be
> excellent?
>
> > > Also, we're not offering "partial workarounds" (excepting some
> > > acknowledged hackery to avoid blocking F31). All of the proposals I
> > > have been discussing in this thread are for real design adjustments
> > > for long-term benefit.
> >
> > I call them "partial" because they are do not address all the issues we are
> > having with Modularity. See my list a few paragraphs above. At least the
> > version conflict issue is not fixable at all without a complete redesign of
> > Modularity that would be incompatible with the current implementation and
> > would probably also violate the FHS (because that is the only way to achieve
> > universal parallel-installability without per-package workarounds).
> >
>
> Parallel-installability is *not* a goal. It is in fact a
> clearly-stated non-goal. I don't see this changing. Package conflicts
> are not a new thing with Modularity.
What about for example Python & other language runtimes that support parallel installation
just fine ? Sure, not everything suports it but needed to have separate module for
each version of Python (and any other piece of software that supports parallel installability)
seems far from optimal.
>
> > And I call them "workarounds" because they add more and more complexity when
> > there would be a simple fix: just don't use default streams anymore. That
> > change might also need short-term workarounds for the upgrade path, but
> > those can all be dropped once the default streams are gone.
> >
>
> It's a matter of perspective: from someone who prefers the classic
> packaging style, you're proposing a fix. From my perspective where
> Modules are a new technology with warts that need to be addressed,
> reverting here is the workaround. Again, that doesn't mean "no", just
> that it's not as clear-cut as you think it is.
>
> > > And while they add some additional complexity on the *infrastructure*, a
> > > primary goal is to not make the users or packagers' lives harder.
> >
> > Yet that is exactly what default streams are doing.
> >
>
> I'm not disagreeing that the current state is bad. See the literal
> next sentence in my reply.
>
> > > We *know* that the default stream/buildroot issue is failing to hit this
> > > goal and the solution is known, implemented upstream and could be deployed
> > > by the end of the week if FESCo gives its approval.
> >
> > The buildroot workaround does not address any of the issues the end users
> > are hitting. You are proposing a different workaround for the upgrade path
> > issue for end users, workaround which as far as I can tell is not
> > implemented yet, which will not address the other issues, and which will
> > still lead to a more complex feel than the simple fix of no longer using
> > default streams.
>
> Now you're moving the goalposts. I addressed the packager concern, so
> you pivoted to attacking our user-experience instead. I acknowledged
> that issue as well as starting this *very thread* to try to find the
> right way to do that. If you don't think the proposal I've made is the
> correct one, suggest how it could be improved instead of simply being
> dismissive.
>
> > I do not understand why you are so strongly against the "no default streams"
> > rule. It would not stop any of the work you are doing on Modularity. You
> > would still be able to ship all your module streams, depend on any other
> > stream of any other module (even if some combinations of streams then
> > conflict), and continue your work as before. The only difference is that
> > instead of making a stream the default, you would merge it into the release
> > branch for the release and build it as a normal "ursine" package. But that
> > does not conflict with any of the Modularity work you are doing.
>
> I'm not strongly against it. I'm hesitant because I don't want to be
> doing reactionary development. We've made mistakes so far by being too
> fast out of the gate. If we're going to change directions
> substantially, I want that to come after careful deliberation.
> Convince me!
>
> > > No, you've decided on the outcome you want to see and have invented a
> > > path to get there that doesn't align with the realities of the
> > > present.
> >
> > Funnily, that is exactly how I would describe what has happened with
> > Modularity so far.
>
> Again, this was a result of being overtired. I apologize for the tone here.
>
> On Wed, Oct 16, 2019 at 9:39 PM Kevin Kofler <kevin.kofler@xxxxxxxxx> wrote:
> > Stephen Gallagher wrote:
> > > On Wed, Oct 16, 2019 at 7:58 PM Kevin Kofler wrote:
> > > > So completely disable all module support in DNF by default with some
> > > > global flag (make all the module code conditional under some new
> > > > enable_modules flag and default the flag to enable_modules = 0), then it
> > > > will treat the packages as normal packages and you only have to provide a
> > > > higher EVR. All this module processing should only happen if the user
> > > > explicitly enables it.
> > >
> > > Given that many of the modules in the distribution currently are there
> > > specifically to provide newer, less-stable versions of the versions in
> > > the non-modular/default stream, this would be fairly disastrous. For
> > > example, Node.js 10.x is the default stream in Fedora 30 because it's
> > > the LTS branch. It also provides a non-default stream for 8.x (the
> > > previous LTS branch that a lot of applications still rely on) and
> > > 12.x, which will be the next LTS branch in November, but is not
> > > guaranteed stable yet. With the approach you're describing, everyone
> > > would be forcibly updated to the unstable 12.x release.
> > >
> > > The only way we could assure ourselves that this wouldn't happen would
> > > be to do another mass-rebuild, bumping the epoch of every package that
> > > exists in a module. That's a lot of work.
> >
> > So it looks like I did not describe clearly enough what my proposed
> > enable_modules=0 flag would do. ("Disable all module code" was apparently
> > too vague.)
> >
> > How I think it should work would be:
> > * For repositories, it completely ignores modular metadata and processes
> > only the non-modular parts of the repository metadata. Therefore, it does
> > not see the Node.js 12.x stream at all. It only sees whatever Node.js is
> > in the non-modular repository. If there are currently only modular
> > versions, then it sees none at all. But with the proposal to require a
> > non-modular default version, it would then see that version, which would
> > likely be Node.js 10.x. Nobody would get forcefully upgraded from 10.x to
> > 12.x.
> > * For installed modules, it completely ignores them, acting as if the
> > database of installed modules were empty. (It just does not read that
> > database at all.)
> > * For installed packages, it treats them all as non-modular. Sure, packages
> > originally installed from a module have weird EVRs encoding module
> > metadata, but otherwise they get processed exactly like a non-modular
> > package. So the default repository only has to provide a newer EVR to
> > upgrade the package. That should address upgrades from 8.x or 10.x to the
> > new default 10.x. If the user had previously installed 12.x, they will
> > only get downgraded if they distro-sync or if package-level dependency
> > issues with the F30 build of 12.x on F31 force the downgrade.
> >
> > I hope that clears it up.
> >
> > > > A slightly more elaborate, but slightly harder to implement, approach
> > > > would be to let DNF simply disable modules that are enabled locally but
> > > > no longer available in the repositories, together with disabling the
> > > > fedora-modular and updates-modular repositories by default.
> > >
> > > And again, this only works if every packager who has spent time
> > > creating a module with a default stream takes their content and shoves
> > > it back into the non-modular repository. Which in some cases they
> > > probably cannot do, because they have build-dependencies that are in
> > > conflict. This is a highly non-trivial process and it would need to be
> > > done individually for every single package. That's far more
> > > packager-hostile than fixing the default stream/buildroot problem and
> > > the upgrade path problem.
> >
> > You are overestimating by far the effort required to demodularize the
> > handful packages that are currently module-only. The evidence Fabio
> > Valentini has gathered so far shows that actually very few packages would be
> > affected and they would not be too hard to fix. And Miro has also offered
> > help with fixing affected packages.
> >
> > All in all, it would require fixing a handful packages once and for all
> > instead of implementing workarounds affecting the entire distribution and
> > its thousands of users.
> >
> > > > And the case of demodularizing packages has to be addressed sooner or
> > > > later anyway, so better address it sooner rather than later, before more
> > > > and more damage is done by maintainers moving packages to module-only
> > > > without a way back.
> > >
> > > I've already acknowledged upthread that demodularizing packages is a
> > > problem we need to solve. It's being worked on, but it's a lot harder
> > > than you think, because we have failsafe code implemented in libdnf to
> > > prevent *accidental* demodularization that's in conflict with this
> > > desire.
> >
> > And exactly that code needs to go, at least in Fedora. I think having a way
> > to migrate away from modules is the common case to prioritize here.
> >
> > That said, it shall be pointed out that, if the proposal to demodularize all
> > default versions of packages gets implemented, we only need a *short term*
> > solution for demodularization in DNF. After 2 releases, we have no default
> > streams left (and they will never come back by policy) and we can expect
> > users to have upgraded through a release with no default streams (given that
> > we do not support upgrading directly to n+3), so at that point DNF can
> > revert to the "safe" behavior (preventing accidental demodularization) by
> > default.
> >
> > So the proposal to demodularize everything could actually make this problem
> > easier to solve.
> >
> > > Also, this paragraph was needlessly antagonistic: moving packages to
> > > modules is not "damage".
> >
> > Sorry for that. The reason I called it "damage" being done is that there is
> > currently no supported way back (as you pointed out yourself) and that it
> > moves us away from the state I (and others, e.g., Miro) believe we should
> > reach (where there are no module-only packages anymore).
> >
> > I consider this approach of making a controversial and experimental change
> > with no contingency plan, then using that absence of a contingency plan as
> > an argument to not only stick to the change at all costs, but even go
> > further with it, entirely unacceptable. (We call it the "creating facts"
> > ("Fakten schaffen") tactic in the German-speaking parts of the world. It is
> > an effective way to bypass discussion and democratic participation.)
> >
> > > Sure, but we are nowhere near a cliff. As I just posted in the Change
> > > Proposal thread, there are three problems we need to solve, two of
> > > which we already have solutions designed for and one (this thread)
> > > that we are trying to finalize. That's far from "standing in front of
> > > a cliff".
> >
> > The problems you listed are not the only ones. There are the version
> > conflicts between non-default streams that can be dragged in by default
> > streams. There is the buildroot-only content, which breaks our self-hosting
> > promise and makes it harder to compile additional software. There is the
> > issue that modules built for an older distribution cannot be kept on a newer
> > distribution version, which is a regression from ursine packages that (e.g.,
> > if they get dropped from the distribution) you can keep as long as they
> > don't depend on an outdated soname, which can be decades. There is the risk
> > that your more and more complex solutions introduce new issues, e.g., bugs
> > and undesirable behavior in DNF, that may or may not be easy to fix. (The
> > more complexity you introduce, the harder it gets for DNF to behave the way
> > the user expects.)
> >
> > And this needs to be brought into relation with the benefits of using
> > default streams instead of non-modular default versions, which as far as I
> > can tell are essentially nonexistent. (Sure, they might be an easier upgrade
> > path for things that are already default streams now, but this is self-
> > referential. We should not stick to default streams forever only because of
> > a historical decision. And actually, the upgrade path from default stream to
> > default stream is also not working yet, which is why we have this thread at
> > all.)
> >
> > > Please understand that "I don't understand how this works" is not the
> > > same thing as "This doesn't work".
> >
> > I understand very well how this doesn't work. :-)
> >
> > > So, literally every word of this is wrong. The negative feedback is
> > > not "overwhelming". It is approximately four noisy individuals, all of
> > > whom have expressed zero interest in understanding the actual
> > > situation that they are trying to "fix" by endlessly insulting the
> > > people working on the problem. Demoralizing the people who can dig us
> > > out of this situation is an unwise strategy.
> >
> > LOL, filing off mailing list consensus as "four noisy individuals". This is
> > getting really ridiculous! Why am I even arguing with somebody who clearly
> > does not want to listen?
> >
> > And you are the people who brought us into this situation to begin with.
> > Default streams should never have been allowed without:
> > 1. an upgrade path from default stream to default stream, AND
> > 2. a contingency plan, i.e., an upgrade path from default stream to
> > non-modular default version
> > The fact that they were implemented without EITHER of these (when actually
> > BOTH are needed) was extremely short-sighted.
> >
> > So bringing yourselves up now as "the people who can dig us out of this
> > situation" feels to me feels like intentionally making a patient sick so you
> > can "cure" them.
> >
> > > Also, we're not offering "partial workarounds" (excepting some
> > > acknowledged hackery to avoid blocking F31). All of the proposals I
> > > have been discussing in this thread are for real design adjustments
> > > for long-term benefit.
> >
> > I call them "partial" because they are do not address all the issues we are
> > having with Modularity. See my list a few paragraphs above. At least the
> > version conflict issue is not fixable at all without a complete redesign of
> > Modularity that would be incompatible with the current implementation and
> > would probably also violate the FHS (because that is the only way to achieve
> > universal parallel-installability without per-package workarounds).
> >
> > And I call them "workarounds" because they add more and more complexity when
> > there would be a simple fix: just don't use default streams anymore. That
> > change might also need short-term workarounds for the upgrade path, but
> > those can all be dropped once the default streams are gone.
> >
> > > And while they add some additional complexity on the *infrastructure*, a
> > > primary goal is to not make the users or packagers' lives harder.
> >
> > Yet that is exactly what default streams are doing.
> >
> > > We *know* that the default stream/buildroot issue is failing to hit this
> > > goal and the solution is known, implemented upstream and could be deployed
> > > by the end of the week if FESCo gives its approval.
> >
> > The buildroot workaround does not address any of the issues the end users
> > are hitting. You are proposing a different workaround for the upgrade path
> > issue for end users, workaround which as far as I can tell is not
> > implemented yet, which will not address the other issues, and which will
> > still lead to a more complex feel than the simple fix of no longer using
> > default streams.
> >
> > I do not understand why you are so strongly against the "no default streams"
> > rule. It would not stop any of the work you are doing on Modularity. You
> > would still be able to ship all your module streams, depend on any other
> > stream of any other module (even if some combinations of streams then
> > conflict), and continue your work as before. The only difference is that
> > instead of making a stream the default, you would merge it into the release
> > branch for the release and build it as a normal "ursine" package. But that
> > does not conflict with any of the Modularity work you are doing.
> >
> > > No, you've decided on the outcome you want to see and have invented a
> > > path to get there that doesn't align with the realities of the
> > > present.
> >
> > Funnily, that is exactly how I would describe what has happened with
> > Modularity so far.
> >
> > Kevin Kofler
> > _______________________________________________
> > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
> > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
> _______________________________________________
> devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
