Re: RFC: Modularity Simplified

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, Nov 26, 2019 at 6:35 PM Zbigniew Jędrzejewski-Szmek
<zbyszek@xxxxxxxxx> wrote:
>
> As a meta-goal, we should break up "Modularity" into a number of
> separate components, some build-side, some client-side. Modularity
> suffers greatly from trying to encode everything into one document.
> This greatly raises the complexity of the task and makes it hard to
> consider alternative proposals for various parts.
>
> Let's consider them separately:
>
> 1. what to build against what branches
>
> You said:
> > * each has package.cfg or some alternative which specifies what is the
> > EOL and/or for which releases to build
>
> Right now we use dist-git branches for this. Having a package.cfg
> might sound easier , but the caveat is that building against different
> branches requires tweaks sometimes. So we immediately hit a choice: do
> we want a single branch and the %ifdef mess, or multiple branches and
> the cherry picking troubles.

Yeah, but if there is clear map between branches & builds stored
somewhere, it could be quite convenient for both. If you say "I want
to have separate branches", then you still have f31/f32 while you
should be able to say "I have 2.x branch and it is going to be built
in all releases". That should be easily discoverable.

> Right now, big disadvantage of multiple branches is that %changelogs
> and Release bumps introduce trivial conflicts. I think we should
> explore the proposals (incl. your and Neal's) to make this better.
> Fixing this would be good also for normal packaging, because it'd remove
> one or two more steps that need to be done every time.

Yeah, at the current job I'm exploring something similar so when I get
some time, I'll try to write proposal.

> We should explore both directions, i.e. package.conf and easier branches.

+1

> We should experiment with better/easier control of what to build
> where. This would benefit normal packaging too. Right now our tooling
> is just hard to use, with multiple commands in different tools needed
> to start builds, schedule updates, control koji status, etc. We need
> changes for "normal" packages as much as for "streams".

+1

> 2. how to deliver rpms to users and how to let them control installed versions
>
> > How I would imagine having multiple streams:
> > * rpms/nodejs has multiple branches - 10, 11, 12
> > * in all of them, nodejs.spec exists as-is without "mangled" name
> > (just with different versions and such)
> > * when builds are submitted, Koji automatically adds suffix to a main
> > package and subpackages containing branch name
> > * during the rpmbuild, extra provides are added (for the unversioned
> > names and indication of a stream) and conflicts (possibly depending on
> > some macro which user can disable) automatically
>
> Yes. The crucial part is that the rpms that are delivered are
> just normal rpms. All the complications are kept on the build-side
> and the client side doesn't need to care if this magic happened, the
> same result could have been achieved with manually mangled spec file.
>
> > * If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can
> > describe this in standard way (Requires: (nodejs >= 9 with nodejs <=
> > 11)) and one will be picked up automatically
> > ...
> > * All standard Conflicts/Requires/Obsoletes/… will simply work
>
> Yes. More of "just normal packages".
>
> > * If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10`
> > * If that requires some conflicting dependency to be switched, it will
> > be switched automatically
> > * Packages produced from nodejs.src have Provides: stream(nodejs)
> > stream(nodejs:9) and Conflicts: stream(nodejs) so that it is
> > explicitly not possible to install 9 and 10 at the same time
> > * If desired, packages can depend on a specific stream via Requires:
> > stream(nodejs:10) without actually depending on nodejs (this requires
> > some small piece of code in libsolv)
> > * In the very similar way, user can lock themselves to a specific
> > stream by writing some conf file which DNF would read (if point above
> > is implemented, about just tens of lines of the code in libdnf, or
> > even libsolv can be teached about that as well)
> > * DNF can be teached about these special things so that you can
> > automatically swap conflicting dependencies, but lock yourself to some
> > streams of a package
>
> ... and here I'm not sure. I'm not convinced we need to try to manage
> all this like that. We already have packages which Conflict, or specify
> conflicting Requires, and the user gives a dnf command, and a certain
> transaction is prepared for them. The transaction will occasionally
> specify a list of rpms to remove. If the user doesn't like this, they can
> cancel and give a different command.

This would imply that we have --allowerasing set by default, but we
don't... And probably we should not since it is dangerous (a bit?).

> Specifically, some streams might conflict, other might not, etc.
> We could have a macro that helps to declare this in one line, but
> let the maintainers of the rpms declare this as needed.
>
>
> 3. [your 2.] Do we want to support buildtime-only packages?
>
> > I would rather generalize this category as "less-supported packages".
> > I maintain 800+ Rust packages and very often I need to update them to
> > an incompatible version. In Rawhide I just do it, update all dependent
> > packages to use new version, and if I can't do that for some reason,
> > create "compat" package. Obviously, all patches are sent to the
> > upstream.
> >[...]
> > * After branching, there will be a script which untags all packages
> > which should not be supported in the new release
> >
> > Upstreams are removing features, I need to deal with Obsoletes but I
> > simply can't continuously add new Obsoletes into the
> > fedora-obsolete-packages. And what for if they are used only during
> > build of other, more important, packages? Why do I have to spend time
> > with upgradepaths? I definitely want some mechanism which will tell to
> > user that "THIS PACKAGE IS NOT FULLY SUPPORTED."
> > Obviously, for packages which are used in runtime need a proper
> > support as we do today for all packages to share work (that's the
> > place where I agree with Kevin Kofler.)
>
> A different proposal: we add a new rpm flag (maybe as a special
> Requires: line) that specifies that a package is only supported as a
> build-time dependency. DNF could then gain a trivial patch that
> it would ask the user "Do you want to install those unsupported packages?
> If you do, please do not file bugs, because ...".
> The advantages:
> a) packages are available for local builds and downstream rebuilds
> b) no problem with mock and koji being different
> c) we avoid any perception of "welding down the engine hood".
>
> This would just acknowledge the truth that there are many packages
> in Fedora that are not supported beyond an occasional version bump
> and rebuild, giving better transparency for the users in general.

+1

> 4.
> > 3. Can we have different lifecycles for the software in Fedora?
> a.k.a. Should we allow more in-release upgrades?
>
> > Right now we have to keep all versions of software which was there at
> > GA point. And from the updates repo. We never remove packages
> > entirely. That said, if package was there at GA time, it will have to
> > be "supported" until the end of that Fedora release.
> > I don't think we can (should?) do much in this regard. If we make
> > packages to build from "stream" branch, we can put an information to
> > that branch that this package should be built for all distributions
> > (Fedora, EPEL) until this date. Or even store this info somewhere else
> > like PDC (yes, I know we want to kill it). fedpkg build will take this
> > into account and submit proper builds. And we can design some API in
> > infra which would tell until when some particular stream of a package
> > is supported. Much like it is done with Fedora releases & GNOME
> > Software.
>
> We can (and do) encode this information directly in dist-git.
> No need to provide a meta-mechanism. Even if package.conf becomes a
> thing, this would contain this information.
>
> I think we should allow "rolling" versions more widely, like we do with
> firefox, the kernel, spam blockers, and probably some other things.
> As long some piece software is most backwards compatible, I think we'd
> be better of abandoning the strict guidelines we have now.
> This depends on each specific package though.

Yeah, the problem I hit with Rust packages that they are not
backwards-compatible. I, for sure, can create compat package but I
would like to have way of telling user "if you have installed this
package, don't expect working upgradepath or anything like that".

> If this happens, it should be as a FPC guideline change, discussed
> and specified independently of the tooling changes. If we detach
> it from the technical side, answer the questions when and if at all
> we allow this will be clearer.
>
>
> 5.
> > 4. Do we want to have some kind of "stream expansion" where software
> > builds against all Pythons, Perls and whatnot?
> >
> > I think this should be conscious choice of a distribution, and in
> > specific cases, maintainer. Using some examples from previous threads,
> > why does bugzilla have to be built against 2 different versions of
> > perl and users could choose? I think maintainer should choose one
> > version of perl and let bugzilla use it. Being able to build
> > combinations of software is definitely nice, but I don't think this
> > should be standard practice. openSUSE does that with ruby and python
> > (they build all modules automatically for all versions) and I like
> > this. But having all packages built against all is just combinatoric
> > explosion. Given how many updates have feedback in Bodhi, I'm pretty
> > sure 99% of combination won't be tested (or even installed?) ever.
>
> Right now this would be prohibitively expensive. But maybe this could
> become much easier if points 1-2 and 4 were better answered.
> So I think we should shelve this question for now.
>
>
> 6. How to deal with packages which need outdated BuildRequires
>
> This is another part of Modularity, and you proposal glosses over this.
> But you said:
> > * With side tags on demand it is now much easier
>
> ... and I fully agree here. There was
> https://fedoraproject.org/wiki/Changes/OnDemandSideTags which plays
> with this idea. I think side-tags could/should be the basis of a
> solution.
>
>
> > 5. Are we still trying to be a Linux distribution or we are just
> > letting people to do whatever they want in our infrastructure?
> > This question bothers me from time to time and I don't have answer.
> > For example, Modularity is very flexible and I very often find people
> > saying that you can expand this and that, but in Fedora we limit its
> > usefulness. Do we actually need to develop something like this
> > (knowing in advance that probably nobody outside of Fedora/RHEL will
> > be using this software / technology)? Are we trying to create
> > technologies which would be very extensible and used in other
> > distributions instead of solving some specific problems? If former,
> > why don't we talk to others about things in advance and not getting
> > other people to work on these cool things?
>
> I don't have an answer for this.
>
> > * fedpkg will be teached about stream branches and will show which
> > branches build where, so that maintainers can easily maintain them
> > * Packaging guidelines should be adopted to accept conflicting
> > packages and tooling should be improved to show the conflicts and how
> > to resolve them
> > * fedora-release will have Suggests: stream(nodejs:10) so that when
> > user types dnf install nodejs, for example nodejs10 would be picked up
> > (instead of nodejs12)
> > * Some macros can be added to generate virtual package with "default
> > stream", so that there will be actual nodejs package depending on
> > nodejs12 if that would be default in given Fedora release
> > * Going further, I would extend Koji with fedora plugin which would
> > deal with the build submission instead of having this logic on clients
> > so that all people get same result
> >
> > ===
> >
> > Now let me quote Matthew Miller with the list of requirements and
> > answer how we achieve them.
> >
> > 1. Users should have alternate streams of software available.
> > I think this is achieved with what I have said above. We will ship
> > nodejs9, nodejs10 at the same time.
> >
> > 2. Those alternate streams should be able to have different lifecycles.
> > nodejs 9 and 10 can automatically build for all distros until EOL
> > (which would be stored somewhere). EOL-date aware tooling (fedpkg)
> > would not build for excluded versions of distros and build for distros
> > where EOL of distro is further than EOL of component.
> >
> > 3. Packaging an individual stream for multiple outputs should be
> > easier than before.
> > As I wrote above, we will have one nodejs rpm repo with multiple
> > branches (aka streams) and no name mangling needed. fedpkg build will
> > spawn builds for all releases matching criteria (EOL date, excluded
> > distributions).
>
> I think all this sounds reasonable. The main part is that it does not
> require massive changes in the koji or dnf, and that it integrates in
> a backwards-compatible way with what we have now.
>
> Let's split the problem into chunks and discuss solutions.

+1, sadly we can't submit hackfest for devconf.cz, but let's probably
meet there with some ideas prepared and discuss them?

> > This is not the exact approach we have tried back in the days, but it
> > is very similar to that. Unfortunately I did not save any documents
> > from those times but Adam or Brian probably still can find them (since
> > they are RH internal).
>
> Zbyszek
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux