-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Modularity is Dead, Long Live Modularity!
=========================================
See this post in glorious technicolor at:
https://communityblog.fedoraproject.org/modularity-dead-long-live-modularity/
Summary
- -------
Fedora’s Modularity initiative aims to make it easy for packagers to
create alternative versions of software and for users to consume those
streams simply. We’ve been working on this for several years, resulting
in the “Boltron” prototype this summer and the recent Fedora Modular
Server beta. Feedback shows that these test releases didn’t meet the
goal, and we’re incorporating that in a modified design which we think
_will_. We plan to demo the new approach by DevConf.cz and FOSDEM.
Retrospective
- -------------
As you have probably read, the members of the Modularity Working Group
and the Server Working Group decided that the current implementation of
Modularity would not meet Fedora’s standard of excellence in time for a
release in Fedora 27 — even with the extra time that the Fedora Council
granted to the project. In particular, feedback received from the
Fedora 27 Modular Server Beta release alerted us to several serious
problems in the design.
The most common feedback from users was: “How do I install package
`foo` in Modular Server?”. In this case, `foo` ranged across a wide
variety of software, including everything from the screen package to
complex third-party applications. In that version of Modularity, a
system was either all Modular or none of it was. To make software
available in the fully-modular system, its packages needed to be part
of a module — and unfortunately, we didn’t succeed in making many of
those.
The lack of available modules was a symptom of a larger problem with
packager involvement. Some of this problem was due to disagreements
with the implementation or design, and some of it was because creating
modules turned out to be more time-consuming and had a steeper learning
curve than we wanted. Additionally, the whole process was not obvious
or well-socialized. The team produced a lot of documentation, but
packagers needed to understand a whole lot of it before even getting
started. As a result, all of the modules slated for delivery in F27
Modular Server were being developed by the Modularity team, a group
that had limited expertise around many of the packages we desired to
ship.
There were other problems, of course. We didn’t really have a strong
plan for how to handle upgrades from traditional deployments. We lacked
a story for how third-party software should be loaded onto the system.
And, we never got away from needing a “bootstrap” module (as explained
below). So, we decided not to ship Modular Server in Fedora 27, and we
activated our fallback plan of releasing Fedora 27 Server Edition in
its classic incarnation at https://getfedora.org/server/.
Moving Forward
- --------------
With all of this in mind, we took Modularity back to the drawing board
(or perhaps “chopping board”). We analyzed the various functionality
that the design offered us and made some trade-offs.
First and foremost, we gave up on the idea of a strictly-maintained,
stable buildroot. Traditional Fedora builds are performed by building
an RPM in a buildroot containing the latest packages available in the
stable updates repository for a release (plus the overrides repo when
needed to allow building against pre-release packages). With
modularity, we hoped that we could define a small and specific
buildroot which would be stable and maintained for the life of a
release. We would then build every module against it. It turned out
when we tried to implement this that it was basically impossible. It
required us to find a point at which the entire buildroot could be used
to build _itself_, a feat which has not been accomplished in Fedora for
many years (if ever). Fedora packages are really only guaranteed to
build successfully against a buildroot for which they have _already_
been built. If it builds against any other buildroot, this is a happy
accident.
There are many reasons for this; something in the dependency chain for
one of the build dependencies may have broken, a bug may have been
introduced in the compiler or linker resulting in an inability to
continue, new default compiler flags may have been added that result in
warnings becoming errors and so on. This was the easiest requirement to
drop as it had never worked. We used a kludgy “bootstrap” module to
work around this which provided a set of packages copied from
non-Modular Fedora to be able to build the platform. In its place, **we
will now build modules against the standard Fedora buildroot**.
Since our buildroot now has access to everything that has been built
for Fedora, we reconsidered another of our original goals that of
providing a module to be the base platform upon which all of the other
modules would depend. Originally, this was intended to define the
provided API of the Base and indicate which portions of it should be
considered stable. However, Fedora already has a stable updates policy
for how things must act within a release. This policy has only a few
outliers (Kernel, KDE, etc.), but for the most part, once we release
the Beta, the platform API is stable for the life of the Fedora
release. By relaxing this requirement, we realized that we could
eliminate (or at least postpone) the requirement for having a defined
platform module.
What we decided instead is to treat Fedora’s “Everything” repository
(essentially, the complete set of software available within a Fedora
release) as the “platform module”, though the tooling will not report
this content as a module. In practical terms, this means that
**creators of modules will no longer need to go through the very
painful process of tracking down which modules provide a dependency
that they need**. Instead, they will be able to depend on the system
version available in the Everything repo. If that version does not meet
their needs, they will have several options described later.
This provides us with several advantages, including **a straightforward
upgrade path from a traditional deployment**, because we will retain
the traditional repositories as well as a set of default modules. This
means it will be possible to upgrade from a current Fedora 27 system to
a modular Fedora 28 system without any special steps. In fact, this
approach will also mean that modularity need not be limited to the
Server Edition.
Another practical advantage to this change is that **module-creation
will become _significantly_ simpler**. Instead of a complex
collection of a package and all of its dependencies, modules will now
only need to describe the parts that differ from the base repository.
For example, Fedora 28 will ship with the Node.js 8.x LTS release in
the standard repository, and a module could be built to provide the 9.x
experimental release as an option. We could also easily provide the
older 6.x LTS release to support older applications. In these cases, we
can ship very simple module definitions which just lists the dist-git
branches matching the desired upstream releases.
In fact, this will now be so simple that **we plan to provide tools to
automate the creation of these modules**. Since most modules will now
only require a single source package be in the components list, we plan
to enable support for automatically building a single branch in
dist-git for all active Fedora (and EPEL) releases. Even for more
complex multi-package modules, the automatically-created module
definitions provide an easy and obvious starting point.
What Will It Look Like?
- -----------------------
>From an end-user’s perspective, **Fedora will ship with two sets of
repositories.** One will be the traditional Fedora repositories
(fedora, updates, and updates-testing) and the other will be a new set
of repositories providing alternative and supplementary modules. We
haven’t decided on a final name for these yet, so we will use the
placeholder terms modular, modular-updates, and
modular-updates-testing.
With this design, **anyon who does not wish to access the additional
versions of software provided by modules can disable the modular and
modular-updates repositories and their system will function exactly as
it does today**. Packages built with Fedora’s traditional process will
be installed and managed from the regular fedora repository, as will
default versions of packages which use the new process behind the
scenes.
**For anyone who wants access to additional versions of packages, these
new module repositories will make them available**. Users will be able
to interact with these new repositories by taking advantage of some new
syntax in DNF, the same as was used in the Modular Server Beta. If a
user wants to install a particular module stream, they can do `dnf
install module[:stream[/profile]]` (e.g. `dnf install @nodejs:6`, which
will install the default profile containing the nodejs and npm
packages). If the user just wants to install whatever version is the
default for this system, a `dnf install package` will continue to work
as it always has.
Conclusion
- ----------
This refined plan offers an understandable, approachable, and
deliverable future for Modularity. Packagers who don’t want to produce
modules will be able to continue packaging exactly as they always have
with **no modification** to their workflows. Those who want to provide
alternative versions of software in a single release or to easily
provide the same version across multiple releases will have new tools
to simplify this.
As the number of available modules grows, users of Fedora will have a
much easier access to the exact version of software they want to
accomplish their tasks. People doing rapid-prototyping can more easily
access newer versions of packages and at the same time people running
older applications can continue to access the older streams that they
need.
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v2.1.0
Comment: https://www.mailvelope.com
wkYEAREIABAFAlo5f/YJEHolVWI2uqOjAAD5RgCfeRL8DcETF34kmCOT5pXJ
yzL0LLYAmwVmmd2DPHf1gzKZOjUfxiHlbuML
=ZnhF
-----END PGP SIGNATURE-----
