On Wed, Oct 16, 2019 at 9:39 PM Kevin Kofler <kevin.kofler@xxxxxxxxx> wrote:
>
> So it looks like I did not describe clearly enough what my proposed
> enable_modules=0 flag would do. ("Disable all module code" was apparently
> too vague.)
>
> How I think it should work would be:
> * For repositories, it completely ignores modular metadata and processes
> only the non-modular parts of the repository metadata. Therefore, it does
> not see the Node.js 12.x stream at all. It only sees whatever Node.js is
> in the non-modular repository. If there are currently only modular
> versions, then it sees none at all. But with the proposal to require a
> non-modular default version, it would then see that version, which would
> likely be Node.js 10.x. Nobody would get forcefully upgraded from 10.x to
> 12.x.
> * For installed modules, it completely ignores them, acting as if the
> database of installed modules were empty. (It just does not read that
> database at all.)
> * For installed packages, it treats them all as non-modular. Sure, packages
> originally installed from a module have weird EVRs encoding module
> metadata, but otherwise they get processed exactly like a non-modular
> package. So the default repository only has to provide a newer EVR to
> upgrade the package. That should address upgrades from 8.x or 10.x to the
> new default 10.x. If the user had previously installed 12.x, they will
> only get downgraded if they distro-sync or if package-level dependency
> issues with the F30 build of 12.x on F31 force the downgrade.
>
> I hope that clears it up.
It does, thanks. I think you're right, that probably *would* work,
though it's slightly harder to do your third bullet point than it
sounds at first blush. I suppose we could fudge it with the
`module_hotfix` option though..
>
> >> A slightly more elaborate, but slightly harder to implement, approach
> >> would be to let DNF simply disable modules that are enabled locally but
> >> no longer available in the repositories, together with disabling the
> >> fedora-modular and updates-modular repositories by default.
> >
> > And again, this only works if every packager who has spent time
> > creating a module with a default stream takes their content and shoves
> > it back into the non-modular repository. Which in some cases they
> > probably cannot do, because they have build-dependencies that are in
> > conflict. This is a highly non-trivial process and it would need to be
> > done individually for every single package. That's far more
> > packager-hostile than fixing the default stream/buildroot problem and
> > the upgrade path problem.
>
> You are overestimating by far the effort required to demodularize the
> handful packages that are currently module-only. The evidence Fabio
> Valentini has gathered so far shows that actually very few packages would be
> affected and they would not be too hard to fix. And Miro has also offered
> help with fixing affected packages.
>
> All in all, it would require fixing a handful packages once and for all
> instead of implementing workarounds affecting the entire distribution and
> its thousands of users.
>
It's worth considering. I'm not ruling it out at this time. I'm not
committed to doing it yet either.
> >> And the case of demodularizing packages has to be addressed sooner or
> >> later anyway, so better address it sooner rather than later, before more
> >> and more damage is done by maintainers moving packages to module-only
> >> without a way back.
> >
> > I've already acknowledged upthread that demodularizing packages is a
> > problem we need to solve. It's being worked on, but it's a lot harder
> > than you think, because we have failsafe code implemented in libdnf to
> > prevent *accidental* demodularization that's in conflict with this
> > desire.
>
> And exactly that code needs to go, at least in Fedora. I think having a way
> to migrate away from modules is the common case to prioritize here.
>
This is one of those cases where I think that RHEL and Fedora needs
are in conflict; in RHEL, we absolutely need to support the failsafe
behavior, because accidentally replacing a critical dependency will
break user applications. In Fedora, this is likely a smaller concern.
It needs investigation.
> That said, it shall be pointed out that, if the proposal to demodularize all
> default versions of packages gets implemented, we only need a *short term*
> solution for demodularization in DNF. After 2 releases, we have no default
> streams left (and they will never come back by policy) and we can expect
> users to have upgraded through a release with no default streams (given that
> we do not support upgrading directly to n+3), so at that point DNF can
> revert to the "safe" behavior (preventing accidental demodularization) by
> default.
>
If we settle on the "no content in default streams" policy for Fedora,
this is a sensible way to go about it, yes.
> So the proposal to demodularize everything could actually make this problem
> easier to solve.
>
> > Also, this paragraph was needlessly antagonistic: moving packages to
> > modules is not "damage".
>
> Sorry for that. The reason I called it "damage" being done is that there is
> currently no supported way back (as you pointed out yourself) and that it
> moves us away from the state I (and others, e.g., Miro) believe we should
> reach (where there are no module-only packages anymore).
>
Yeah, I can understand that perspective.
> I consider this approach of making a controversial and experimental change
> with no contingency plan, then using that absence of a contingency plan as
> an argument to not only stick to the change at all costs, but even go
> further with it, entirely unacceptable. (We call it the "creating facts"
> ("Fakten schaffen") tactic in the German-speaking parts of the world. It is
> an effective way to bypass discussion and democratic participation.)
>
Yeah, I'm not trying to make this a "fait accompli" discussion either.
We are where we are. I just don't want to revert as a knee-jerk
reaction if we can find a sustainable solution. I do understand your
frustrations in the way things have happened thus far. I'm not
entirely convinced that killing off default streams is the right
approach, but you and Miro have made enough compelling arguments that
it has to be considered carefully.
> > Sure, but we are nowhere near a cliff. As I just posted in the Change
> > Proposal thread, there are three problems we need to solve, two of
> > which we already have solutions designed for and one (this thread)
> > that we are trying to finalize. That's far from "standing in front of
> > a cliff".
>
> The problems you listed are not the only ones. There are the version
> conflicts between non-default streams that can be dragged in by default
> streams.
I submit that this is true of the non-modular content as well. We've
worked around this in some places with the alternatives system, but
there are plenty of cases where installing an RPM means that some
other RPMs are now in conflict. This just moves it to another layer.
> There is the buildroot-only content, which breaks our self-hosting
> promise and makes it harder to compile additional software.
We don't really have a self-hosting promise; We have an aspiration. We
know that in reality, if we tried to do a mass-rebuild at Final
Freeze, we would fail to build parts of the OS. We can only assert
that *at some time* we were able to build everything with other
packages available in Fedora. Also, the Ursa Prime Change will allow
us to ship this in the buildroot repository publicly, so I think that
is addressed.
> There is the
> issue that modules built for an older distribution cannot be kept on a newer
> distribution version, which is a regression from ursine packages that (e.g.,
> if they get dropped from the distribution) you can keep as long as they
> don't depend on an outdated soname, which can be decades. There is the risk
> that your more and more complex solutions introduce new issues, e.g., bugs
> and undesirable behavior in DNF, that may or may not be easy to fix. (The
> more complexity you introduce, the harder it gets for DNF to behave the way
> the user expects.)
Would you mind opening a ticket at https://pagure.io/modularity/issues
and/or a new mail thread on this? I think there's value to what you're
suggesting. We need to figure out what the right experience is and I
think this thread is the wrong place for it.
>
> And this needs to be brought into relation with the benefits of using
> default streams instead of non-modular default versions, which as far as I
> can tell are essentially nonexistent. (Sure, they might be an easier upgrade
> path for things that are already default streams now, but this is self-
> referential. We should not stick to default streams forever only because of
> a historical decision. And actually, the upgrade path from default stream to
> default stream is also not working yet, which is why we have this thread at
> all.)
As Alexander pointed out elsewhere in the thread, there *are* other
benefits to being able to customize the buildroot. Some of them can be
emulated with side-tags, buildroot overrides and compat packages, but
that's pretty complex. If we make such a choice, we need to understand
what we are losing.
>
> > Please understand that "I don't understand how this works" is not the
> > same thing as "This doesn't work".
>
> I understand very well how this doesn't work. :-)
>
Touché
> > So, literally every word of this is wrong. The negative feedback is
> > not "overwhelming". It is approximately four noisy individuals, all of
> > whom have expressed zero interest in understanding the actual
> > situation that they are trying to "fix" by endlessly insulting the
> > people working on the problem. Demoralizing the people who can dig us
> > out of this situation is an unwise strategy.
>
> LOL, filing off mailing list consensus as "four noisy individuals". This is
> getting really ridiculous! Why am I even arguing with somebody who clearly
> does not want to listen?
>
See my other reply. This was me getting frustrated while being
overtired. Please accept my apology for that outburst.
> And you are the people who brought us into this situation to begin with.
> Default streams should never have been allowed without:
> 1. an upgrade path from default stream to default stream, AND
This was a lack of foresight. We didn't account for this and didn't
realize it was a problem until users started complaining that they
expected it. We can't think of everything.
> 2. a contingency plan, i.e., an upgrade path from default stream to
> non-modular default version
Pretty much the same answer as above. It's easy with hindsight to say
these were obvious and expected, but we are only human.
> The fact that they were implemented without EITHER of these (when actually
> BOTH are needed) was extremely short-sighted.
>
> So bringing yourselves up now as "the people who can dig us out of this
> situation" feels to me feels like intentionally making a patient sick so you
> can "cure" them.
>
I think that's a little harsh (but probably fair given my tone above).
Can we agree that we're both on the same side: we want Fedora to be
excellent?
> > Also, we're not offering "partial workarounds" (excepting some
> > acknowledged hackery to avoid blocking F31). All of the proposals I
> > have been discussing in this thread are for real design adjustments
> > for long-term benefit.
>
> I call them "partial" because they are do not address all the issues we are
> having with Modularity. See my list a few paragraphs above. At least the
> version conflict issue is not fixable at all without a complete redesign of
> Modularity that would be incompatible with the current implementation and
> would probably also violate the FHS (because that is the only way to achieve
> universal parallel-installability without per-package workarounds).
>
Parallel-installability is *not* a goal. It is in fact a
clearly-stated non-goal. I don't see this changing. Package conflicts
are not a new thing with Modularity.
> And I call them "workarounds" because they add more and more complexity when
> there would be a simple fix: just don't use default streams anymore. That
> change might also need short-term workarounds for the upgrade path, but
> those can all be dropped once the default streams are gone.
>
It's a matter of perspective: from someone who prefers the classic
packaging style, you're proposing a fix. From my perspective where
Modules are a new technology with warts that need to be addressed,
reverting here is the workaround. Again, that doesn't mean "no", just
that it's not as clear-cut as you think it is.
> > And while they add some additional complexity on the *infrastructure*, a
> > primary goal is to not make the users or packagers' lives harder.
>
> Yet that is exactly what default streams are doing.
>
I'm not disagreeing that the current state is bad. See the literal
next sentence in my reply.
> > We *know* that the default stream/buildroot issue is failing to hit this
> > goal and the solution is known, implemented upstream and could be deployed
> > by the end of the week if FESCo gives its approval.
>
> The buildroot workaround does not address any of the issues the end users
> are hitting. You are proposing a different workaround for the upgrade path
> issue for end users, workaround which as far as I can tell is not
> implemented yet, which will not address the other issues, and which will
> still lead to a more complex feel than the simple fix of no longer using
> default streams.
Now you're moving the goalposts. I addressed the packager concern, so
you pivoted to attacking our user-experience instead. I acknowledged
that issue as well as starting this *very thread* to try to find the
right way to do that. If you don't think the proposal I've made is the
correct one, suggest how it could be improved instead of simply being
dismissive.
>
> I do not understand why you are so strongly against the "no default streams"
> rule. It would not stop any of the work you are doing on Modularity. You
> would still be able to ship all your module streams, depend on any other
> stream of any other module (even if some combinations of streams then
> conflict), and continue your work as before. The only difference is that
> instead of making a stream the default, you would merge it into the release
> branch for the release and build it as a normal "ursine" package. But that
> does not conflict with any of the Modularity work you are doing.
I'm not strongly against it. I'm hesitant because I don't want to be
doing reactionary development. We've made mistakes so far by being too
fast out of the gate. If we're going to change directions
substantially, I want that to come after careful deliberation.
Convince me!
>
> > No, you've decided on the outcome you want to see and have invented a
> > path to get there that doesn't align with the realities of the
> > present.
>
> Funnily, that is exactly how I would describe what has happened with
> Modularity so far.
Again, this was a result of being overtired. I apologize for the tone here.
On Wed, Oct 16, 2019 at 9:39 PM Kevin Kofler <kevin.kofler@xxxxxxxxx> wrote:
>
> Stephen Gallagher wrote:
> > On Wed, Oct 16, 2019 at 7:58 PM Kevin Kofler wrote:
> >> So completely disable all module support in DNF by default with some
> >> global flag (make all the module code conditional under some new
> >> enable_modules flag and default the flag to enable_modules = 0), then it
> >> will treat the packages as normal packages and you only have to provide a
> >> higher EVR. All this module processing should only happen if the user
> >> explicitly enables it.
> >
> > Given that many of the modules in the distribution currently are there
> > specifically to provide newer, less-stable versions of the versions in
> > the non-modular/default stream, this would be fairly disastrous. For
> > example, Node.js 10.x is the default stream in Fedora 30 because it's
> > the LTS branch. It also provides a non-default stream for 8.x (the
> > previous LTS branch that a lot of applications still rely on) and
> > 12.x, which will be the next LTS branch in November, but is not
> > guaranteed stable yet. With the approach you're describing, everyone
> > would be forcibly updated to the unstable 12.x release.
> >
> > The only way we could assure ourselves that this wouldn't happen would
> > be to do another mass-rebuild, bumping the epoch of every package that
> > exists in a module. That's a lot of work.
>
> So it looks like I did not describe clearly enough what my proposed
> enable_modules=0 flag would do. ("Disable all module code" was apparently
> too vague.)
>
> How I think it should work would be:
> * For repositories, it completely ignores modular metadata and processes
> only the non-modular parts of the repository metadata. Therefore, it does
> not see the Node.js 12.x stream at all. It only sees whatever Node.js is
> in the non-modular repository. If there are currently only modular
> versions, then it sees none at all. But with the proposal to require a
> non-modular default version, it would then see that version, which would
> likely be Node.js 10.x. Nobody would get forcefully upgraded from 10.x to
> 12.x.
> * For installed modules, it completely ignores them, acting as if the
> database of installed modules were empty. (It just does not read that
> database at all.)
> * For installed packages, it treats them all as non-modular. Sure, packages
> originally installed from a module have weird EVRs encoding module
> metadata, but otherwise they get processed exactly like a non-modular
> package. So the default repository only has to provide a newer EVR to
> upgrade the package. That should address upgrades from 8.x or 10.x to the
> new default 10.x. If the user had previously installed 12.x, they will
> only get downgraded if they distro-sync or if package-level dependency
> issues with the F30 build of 12.x on F31 force the downgrade.
>
> I hope that clears it up.
>
> >> A slightly more elaborate, but slightly harder to implement, approach
> >> would be to let DNF simply disable modules that are enabled locally but
> >> no longer available in the repositories, together with disabling the
> >> fedora-modular and updates-modular repositories by default.
> >
> > And again, this only works if every packager who has spent time
> > creating a module with a default stream takes their content and shoves
> > it back into the non-modular repository. Which in some cases they
> > probably cannot do, because they have build-dependencies that are in
> > conflict. This is a highly non-trivial process and it would need to be
> > done individually for every single package. That's far more
> > packager-hostile than fixing the default stream/buildroot problem and
> > the upgrade path problem.
>
> You are overestimating by far the effort required to demodularize the
> handful packages that are currently module-only. The evidence Fabio
> Valentini has gathered so far shows that actually very few packages would be
> affected and they would not be too hard to fix. And Miro has also offered
> help with fixing affected packages.
>
> All in all, it would require fixing a handful packages once and for all
> instead of implementing workarounds affecting the entire distribution and
> its thousands of users.
>
> >> And the case of demodularizing packages has to be addressed sooner or
> >> later anyway, so better address it sooner rather than later, before more
> >> and more damage is done by maintainers moving packages to module-only
> >> without a way back.
> >
> > I've already acknowledged upthread that demodularizing packages is a
> > problem we need to solve. It's being worked on, but it's a lot harder
> > than you think, because we have failsafe code implemented in libdnf to
> > prevent *accidental* demodularization that's in conflict with this
> > desire.
>
> And exactly that code needs to go, at least in Fedora. I think having a way
> to migrate away from modules is the common case to prioritize here.
>
> That said, it shall be pointed out that, if the proposal to demodularize all
> default versions of packages gets implemented, we only need a *short term*
> solution for demodularization in DNF. After 2 releases, we have no default
> streams left (and they will never come back by policy) and we can expect
> users to have upgraded through a release with no default streams (given that
> we do not support upgrading directly to n+3), so at that point DNF can
> revert to the "safe" behavior (preventing accidental demodularization) by
> default.
>
> So the proposal to demodularize everything could actually make this problem
> easier to solve.
>
> > Also, this paragraph was needlessly antagonistic: moving packages to
> > modules is not "damage".
>
> Sorry for that. The reason I called it "damage" being done is that there is
> currently no supported way back (as you pointed out yourself) and that it
> moves us away from the state I (and others, e.g., Miro) believe we should
> reach (where there are no module-only packages anymore).
>
> I consider this approach of making a controversial and experimental change
> with no contingency plan, then using that absence of a contingency plan as
> an argument to not only stick to the change at all costs, but even go
> further with it, entirely unacceptable. (We call it the "creating facts"
> ("Fakten schaffen") tactic in the German-speaking parts of the world. It is
> an effective way to bypass discussion and democratic participation.)
>
> > Sure, but we are nowhere near a cliff. As I just posted in the Change
> > Proposal thread, there are three problems we need to solve, two of
> > which we already have solutions designed for and one (this thread)
> > that we are trying to finalize. That's far from "standing in front of
> > a cliff".
>
> The problems you listed are not the only ones. There are the version
> conflicts between non-default streams that can be dragged in by default
> streams. There is the buildroot-only content, which breaks our self-hosting
> promise and makes it harder to compile additional software. There is the
> issue that modules built for an older distribution cannot be kept on a newer
> distribution version, which is a regression from ursine packages that (e.g.,
> if they get dropped from the distribution) you can keep as long as they
> don't depend on an outdated soname, which can be decades. There is the risk
> that your more and more complex solutions introduce new issues, e.g., bugs
> and undesirable behavior in DNF, that may or may not be easy to fix. (The
> more complexity you introduce, the harder it gets for DNF to behave the way
> the user expects.)
>
> And this needs to be brought into relation with the benefits of using
> default streams instead of non-modular default versions, which as far as I
> can tell are essentially nonexistent. (Sure, they might be an easier upgrade
> path for things that are already default streams now, but this is self-
> referential. We should not stick to default streams forever only because of
> a historical decision. And actually, the upgrade path from default stream to
> default stream is also not working yet, which is why we have this thread at
> all.)
>
> > Please understand that "I don't understand how this works" is not the
> > same thing as "This doesn't work".
>
> I understand very well how this doesn't work. :-)
>
> > So, literally every word of this is wrong. The negative feedback is
> > not "overwhelming". It is approximately four noisy individuals, all of
> > whom have expressed zero interest in understanding the actual
> > situation that they are trying to "fix" by endlessly insulting the
> > people working on the problem. Demoralizing the people who can dig us
> > out of this situation is an unwise strategy.
>
> LOL, filing off mailing list consensus as "four noisy individuals". This is
> getting really ridiculous! Why am I even arguing with somebody who clearly
> does not want to listen?
>
> And you are the people who brought us into this situation to begin with.
> Default streams should never have been allowed without:
> 1. an upgrade path from default stream to default stream, AND
> 2. a contingency plan, i.e., an upgrade path from default stream to
> non-modular default version
> The fact that they were implemented without EITHER of these (when actually
> BOTH are needed) was extremely short-sighted.
>
> So bringing yourselves up now as "the people who can dig us out of this
> situation" feels to me feels like intentionally making a patient sick so you
> can "cure" them.
>
> > Also, we're not offering "partial workarounds" (excepting some
> > acknowledged hackery to avoid blocking F31). All of the proposals I
> > have been discussing in this thread are for real design adjustments
> > for long-term benefit.
>
> I call them "partial" because they are do not address all the issues we are
> having with Modularity. See my list a few paragraphs above. At least the
> version conflict issue is not fixable at all without a complete redesign of
> Modularity that would be incompatible with the current implementation and
> would probably also violate the FHS (because that is the only way to achieve
> universal parallel-installability without per-package workarounds).
>
> And I call them "workarounds" because they add more and more complexity when
> there would be a simple fix: just don't use default streams anymore. That
> change might also need short-term workarounds for the upgrade path, but
> those can all be dropped once the default streams are gone.
>
> > And while they add some additional complexity on the *infrastructure*, a
> > primary goal is to not make the users or packagers' lives harder.
>
> Yet that is exactly what default streams are doing.
>
> > We *know* that the default stream/buildroot issue is failing to hit this
> > goal and the solution is known, implemented upstream and could be deployed
> > by the end of the week if FESCo gives its approval.
>
> The buildroot workaround does not address any of the issues the end users
> are hitting. You are proposing a different workaround for the upgrade path
> issue for end users, workaround which as far as I can tell is not
> implemented yet, which will not address the other issues, and which will
> still lead to a more complex feel than the simple fix of no longer using
> default streams.
>
> I do not understand why you are so strongly against the "no default streams"
> rule. It would not stop any of the work you are doing on Modularity. You
> would still be able to ship all your module streams, depend on any other
> stream of any other module (even if some combinations of streams then
> conflict), and continue your work as before. The only difference is that
> instead of making a stream the default, you would merge it into the release
> branch for the release and build it as a normal "ursine" package. But that
> does not conflict with any of the Modularity work you are doing.
>
> > No, you've decided on the outcome you want to see and have invented a
> > path to get there that doesn't align with the realities of the
> > present.
>
> Funnily, that is exactly how I would describe what has happened with
> Modularity so far.
>
> Kevin Kofler
> _______________________________________________
> devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
