Re: Modularity and the system-upgrade path

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Stephen Gallagher wrote:
> On Wed, Oct 16, 2019 at 7:58 PM Kevin Kofler wrote:
>> So completely disable all module support in DNF by default with some
>> global flag (make all the module code conditional under some new
>> enable_modules flag and default the flag to enable_modules = 0), then it
>> will treat the packages as normal packages and you only have to provide a
>> higher EVR. All this module processing should only happen if the user
>> explicitly enables it.
>
> Given that many of the modules in the distribution currently are there
> specifically to provide newer, less-stable versions of the versions in
> the non-modular/default stream, this would be fairly disastrous. For
> example, Node.js 10.x is the default stream in Fedora 30 because it's
> the LTS branch. It also provides a non-default stream for 8.x (the
> previous LTS branch that a lot of applications still rely on) and
> 12.x, which will be the next LTS branch in November, but is not
> guaranteed stable yet. With the approach you're describing, everyone
> would be forcibly updated to the unstable 12.x release.
> 
> The only way we could assure ourselves that this wouldn't happen would
> be to do another mass-rebuild, bumping the epoch of every package that
> exists in a module. That's a lot of work.

So it looks like I did not describe clearly enough what my proposed 
enable_modules=0 flag would do. ("Disable all module code" was apparently 
too vague.)

How I think it should work would be:
* For repositories, it completely ignores modular metadata and processes
  only the non-modular parts of the repository metadata. Therefore, it does
  not see the Node.js 12.x stream at all. It only sees whatever Node.js is
  in the non-modular repository. If there are currently only modular
  versions, then it sees none at all. But with the proposal to require a
  non-modular default version, it would then see that version, which would
  likely be Node.js 10.x. Nobody would get forcefully upgraded from 10.x to
  12.x.
* For installed modules, it completely ignores them, acting as if the
  database of installed modules were empty. (It just does not read that
  database at all.)
* For installed packages, it treats them all as non-modular. Sure, packages
  originally installed from a module have weird EVRs encoding module
  metadata, but otherwise they get processed exactly like a non-modular
  package. So the default repository only has to provide a newer EVR to
  upgrade the package. That should address upgrades from 8.x or 10.x to the
  new default 10.x. If the user had previously installed 12.x, they will
  only get downgraded if they distro-sync or if package-level dependency
  issues with the F30 build of 12.x on F31 force the downgrade.

I hope that clears it up.

>> A slightly more elaborate, but slightly harder to implement, approach
>> would be to let DNF simply disable modules that are enabled locally but
>> no longer available in the repositories, together with disabling the
>> fedora-modular and updates-modular repositories by default.
> 
> And again, this only works if every packager who has spent time
> creating a module with a default stream takes their content and shoves
> it back into the non-modular repository. Which in some cases they
> probably cannot do, because they have build-dependencies that are in
> conflict. This is a highly non-trivial process and it would need to be
> done individually for every single package. That's far more
> packager-hostile than fixing the default stream/buildroot problem and
> the upgrade path problem.

You are overestimating by far the effort required to demodularize the 
handful packages that are currently module-only. The evidence Fabio 
Valentini has gathered so far shows that actually very few packages would be 
affected and they would not be too hard to fix. And Miro has also offered 
help with fixing affected packages.

All in all, it would require fixing a handful packages once and for all 
instead of implementing workarounds affecting the entire distribution and 
its thousands of users.

>> And the case of demodularizing packages has to be addressed sooner or
>> later anyway, so better address it sooner rather than later, before more
>> and more damage is done by maintainers moving packages to module-only
>> without a way back.
> 
> I've already acknowledged upthread that demodularizing packages is a
> problem we need to solve. It's being worked on, but it's a lot harder
> than you think, because we have failsafe code implemented in libdnf to
> prevent *accidental* demodularization that's in conflict with this
> desire.

And exactly that code needs to go, at least in Fedora. I think having a way 
to migrate away from modules is the common case to prioritize here.

That said, it shall be pointed out that, if the proposal to demodularize all 
default versions of packages gets implemented, we only need a *short term* 
solution for demodularization in DNF. After 2 releases, we have no default 
streams left (and they will never come back by policy) and we can expect 
users to have upgraded through a release with no default streams (given that 
we do not support upgrading directly to n+3), so at that point DNF can 
revert to the "safe" behavior (preventing accidental demodularization) by 
default.

So the proposal to demodularize everything could actually make this problem 
easier to solve.

> Also, this paragraph was needlessly antagonistic: moving packages to
> modules is not "damage".

Sorry for that. The reason I called it "damage" being done is that there is 
currently no supported way back (as you pointed out yourself) and that it 
moves us away from the state I (and others, e.g., Miro) believe we should 
reach (where there are no module-only packages anymore).

I consider this approach of making a controversial and experimental change 
with no contingency plan, then using that absence of a contingency plan as 
an argument to not only stick to the change at all costs, but even go 
further with it, entirely unacceptable. (We call it the "creating facts" 
("Fakten schaffen") tactic in the German-speaking parts of the world. It is 
an effective way to bypass discussion and democratic participation.)

> Sure, but we are nowhere near a cliff. As I just posted in the Change
> Proposal thread, there are three problems we need to solve, two of
> which we already have solutions designed for and one (this thread)
> that we are trying to finalize. That's far from "standing in front of
> a cliff".

The problems you listed are not the only ones. There are the version 
conflicts between non-default streams that can be dragged in by default 
streams. There is the buildroot-only content, which breaks our self-hosting 
promise and makes it harder to compile additional software. There is the 
issue that modules built for an older distribution cannot be kept on a newer 
distribution version, which is a regression from ursine packages that (e.g., 
if they get dropped from the distribution) you can keep as long as they 
don't depend on an outdated soname, which can be decades. There is the risk 
that your more and more complex solutions introduce new issues, e.g., bugs 
and undesirable behavior in DNF, that may or may not be easy to fix. (The 
more complexity you introduce, the harder it gets for DNF to behave the way 
the user expects.)

And this needs to be brought into relation with the benefits of using 
default streams instead of non-modular default versions, which as far as I 
can tell are essentially nonexistent. (Sure, they might be an easier upgrade 
path for things that are already default streams now, but this is self-
referential. We should not stick to default streams forever only because of 
a historical decision. And actually, the upgrade path from default stream to 
default stream is also not working yet, which is why we have this thread at 
all.)

> Please understand that "I don't understand how this works" is not the
> same thing as "This doesn't work".

I understand very well how this doesn't work. :-)

> So, literally every word of this is wrong. The negative feedback is
> not "overwhelming". It is approximately four noisy individuals, all of
> whom have expressed zero interest in understanding the actual
> situation that they are trying to "fix" by endlessly insulting the
> people working on the problem. Demoralizing the people who can dig us
> out of this situation is an unwise strategy.

LOL, filing off mailing list consensus as "four noisy individuals". This is 
getting really ridiculous! Why am I even arguing with somebody who clearly 
does not want to listen?

And you are the people who brought us into this situation to begin with. 
Default streams should never have been allowed without:
1. an upgrade path from default stream to default stream, AND
2. a contingency plan, i.e., an upgrade path from default stream to
   non-modular default version
The fact that they were implemented without EITHER of these (when actually 
BOTH are needed) was extremely short-sighted.

So bringing yourselves up now as "the people who can dig us out of this 
situation" feels to me feels like intentionally making a patient sick so you 
can "cure" them.

> Also, we're not offering "partial workarounds" (excepting some
> acknowledged hackery to avoid blocking F31). All of the proposals I
> have been discussing in this thread are for real design adjustments
> for long-term benefit.

I call them "partial" because they are do not address all the issues we are 
having with Modularity. See my list a few paragraphs above. At least the 
version conflict issue is not fixable at all without a complete redesign of 
Modularity that would be incompatible with the current implementation and 
would probably also violate the FHS (because that is the only way to achieve 
universal parallel-installability without per-package workarounds).

And I call them "workarounds" because they add more and more complexity when 
there would be a simple fix: just don't use default streams anymore. That 
change might also need short-term workarounds for the upgrade path, but 
those can all be dropped once the default streams are gone.

> And while they add some additional complexity on the *infrastructure*, a
> primary goal is to not make the users or packagers' lives harder.

Yet that is exactly what default streams are doing.

> We *know* that the default stream/buildroot issue is failing to hit this
> goal and the solution is known, implemented upstream and could be deployed
> by the end of the week if FESCo gives its approval.

The buildroot workaround does not address any of the issues the end users 
are hitting. You are proposing a different workaround for the upgrade path 
issue for end users, workaround which as far as I can tell is not 
implemented yet, which will not address the other issues, and which will 
still lead to a more complex feel than the simple fix of no longer using 
default streams.

I do not understand why you are so strongly against the "no default streams" 
rule. It would not stop any of the work you are doing on Modularity. You 
would still be able to ship all your module streams, depend on any other 
stream of any other module (even if some combinations of streams then 
conflict), and continue your work as before. The only difference is that 
instead of making a stream the default, you would merge it into the release 
branch for the release and build it as a normal "ursine" package. But that 
does not conflict with any of the Modularity work you are doing.

> No, you've decided on the outcome you want to see and have invented a
> path to get there that doesn't align with the realities of the
> present.

Funnily, that is exactly how I would describe what has happened with 
Modularity so far.

        Kevin Kofler
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux