This is a draft of the proposal I'm presenting at Flock, "An Architecture
for a More Agile Fedora" (<http://sched.co/19ugKGM>). It represents a big
change to how we as the Fedora project put together our distribution. I've
gone through several drafts of this and talked to a few people, and I'd
really value all of your feedback to refine it further.
Preliminary slides are available at <http://mattdm.org/fedora/next/>; I will
update them as this discussion progresses. (So, page numbers might change.)
What follows is an annotated text version of the presentation — the
annotations basically being what I might say as each slide displays.
If you have a short attention span, you might want flip through the slides
first and come back for explanation where it's unclear. And the explanation
here might be unclear too — please ask questions.
An Architecture for a More Agile Fedora
presented by Matthew Miller ( ← with input from many people)
===
Problem Statement (What's this all about?)
---
Let's Start with What's Good
---
* Fedora is awesome
* Great tech, great people
* Amazing, high standards
* Best enthusiast distro
* Solid base for RHEL
* Pretty decent cloud image
I've been involved to some small degree with Fedora since the beginning.
This is a project and a Linux distribution which I love, and which I know
everyone reading this also cares about deeply. We don't want to break what
is successful and works.
But...
---
* We need to be more than that
* Not widely used by RHEL users
* Not so great for building on...
* Including Red Hat's own stuff
* We're not seen as relevant...
* Let alone exciting
Whenever I go to a tech meetup or talk to someone from a new startup
company, their developers are inevitably using a different (usually
proprietary) desktop OS, plus a non-Fedora distribution on their code. We're
being left behind and left out. It doesn't matter how theoretically great we
are if we end up with no users.
The Idea
---
Focus core distro as platform, include layers of modular enabling
technologies, and provide room for special interest groups to create
solutions within the Fedora circle.
Starting with Ring One
---
* We draw a line around a small "base design"
* E.g.: Everything in @standard, plus the toolchains to build it
* Line could go elsewhere; that's an implementation detail
* Most strict change management
* Packaging standards as current Fedora
The example line is somewhat arbitrary here — I'll talk a little more about
practicalities at the end of the talk. The idea of using @standard as a
starting place is just that. That's the group that's one step up from a
minimal install; common things usually installed. That collection of
packages is arbitrary and has grown arbitrarily, but it makes a place to
begin. We'll probably end up with a set of packages that isn't exactly tied
to any existing group.
Anyway, this is the core OS. It's bigger than a minimal install, and is a
complete but lightweight Linux distribution. It has a very clear charter:
use the latest in open source platform-level software integrated into a
solid foundation for the higher rings.
The Base OS
---
* Ring One is base functionality and behavior that everyone can expect of any Fedora system
* Making a distinction helps focus engineering resources
* And it makes a foundation that higher levels know they can trust
Ring 1 : A Picture
---
*********
** **
** **
* Ring 1 *
* *
* Fedora Core *
** **
** **
*********
Wait, did you just say Fedora _Core_?
Yes I did.
But, this is not a return to the old Core + Extras, because the line is
drawn based on _what_ and _how_ rather than on who works for what company.
There were two big problems with the long-ago approach.
First, the "core" was developed internally and not as a community open
source project. If a package was in that part of the distribution, there was
no real route for participation. Bad.
Second, the quality standards for Fedora Extras, the collection of packages
built around the core, were much, much higher. Upside-down!
So, we won't repeat those mistakes. The new "Fedora Core" is a community
project just like Fedora today. And we focus the most strict standards on
the inner rings.
Ring Zero: Fedora Minimal
---
* Start from the current @core, and aim to shrink
* Intentional focus on minimalism
* JEOS — Just Enough Operating System
* Even stronger rules than Ring 1
* Not self-hosting
This is pretty self-explanatory. There is a lot of demand and many use cases
for a "Just Enough Fedora" operating system. Ring 0 exists to provide that
in a serious way which Fedora has never really covered before.
Rings 1 + 0: Together
---
(There is a diagram here, but I'm leaving it out of the text. It's
a circle. With another circle.)
Part Two... First, _Why?_
---
* Back to the problem statement: we need to be useful
* People building on Fedora need a reliable base...
* ... but need more flexibility in the areas relevant to their code
* So let's design something that gives that
***DON'T PANIC***
This is a vision for the future. The Fedora Core idea can get started now,
and the ideas beyond that are for development over a longer term. That
doesn't mean never, and there are some specific things at the higher level
to get started on now, but I'm not suggesting to change everything all
crazily. We're not throwing out what have. The idea is to explore several
different approaches which will better position Fedora as the distro of
choice for future development.
So... Rings 2 + 1 + 0
---
(There is another diagram here. It's more circles inside of circles.)
Ring Two: Environments and Stacks
---
* Environments are where you run the code you care about
* Stacks are modular collections of _software which is used by other
software_
In one of the earlier drafts, I had these as individual, separate rings.
That turns out to be kind of distracting, because depending on your point
of view either could easily be seen as inside or outside the other. So, I
put them together in this level.
Environments
---
* Desktop environments, ideally with app containers
* Platform as a Service, decoupled: integrated OpenShift Gears
* Core Image (oVirt node, cloud guest)
Stacks
---
* Languages, databases, libraries
* Maybe X/Wayland
* The kind of things which could be OpenShift Cartridges or Software
Collections
(For Example)
---
And there's another diagram here. In the Ring 2 circle, we have examples
like Perl; Ruby 1.8.x, 1.9.x, 2.0; Python 2 & 3; Java, Wildfly; some
databases, desktop environments, OpenShift, and then something called Fedora
Commons, which I'll talk about in a bit.
What's a Ring?
---
* Rings give SIGs a way to replace the default expectations with their own
* Separate policies may apply to each ring
* But also, also, we need community, policies, and infrastructure to make
them _part of Fedora_
So, there's nothing really magical about the circles. The main idea is that
we need to make room for more exploration within Fedora beyond the
traditional idea of how a Linux distribution is put together, and this is a
model for describing how and where that can work.
We still need global Fedora policies, but they'll apply differently to each
ring. In Ring 2, SIGs will be given a lot of room to develop their own
rules, but in Ring 1, not so much. Different environments and stacks within
Ring 2 may have different individual policies from each other, but there
will also be general policies for Ring 2 which are different from those for
Ring 1, and of course global policies that apply to the whole thing (e.g.,
no proprietary code).
Possible Examples
---
* Possible example: bundled libs okay, but must be documented
* Can override versions of software at lower tier
* Can overlap with software from other stacks
* Not necessarily even RPM
* Different lifecycle (but shared release cadence)
Big stuff here.
Obviously, no-bundled-libs is a crucial part of the packaging guidelines
today. As a sysadmin, I know why it's important. This is not just a noble
goal, but also something that pragmatically makes systems better. But, it's
also keeping us from having software that people really use in Fedora. Chef
and Hadoop are two big examples. This hurts us more than it helps the world.
So, in some areas, we need a different approach.
Overriding other parts of the distribution is crucial, and I'll talk a
little bit more about different approaches and what they mean in a little
bit.
Some approaches may even move beyond RPM packaging. This can mean focusing
on "language native" package formats like Jars, Gems, and Eggs; or it can
mean moving to a Git-based distribution model for some parts of the distro.
And, different lifecycle: a SIG may decide that Ruby 1.9.x support should
continue for some number of years and maintain that stack against different
Fedora Core releases. We do this already in some ways, but having a clear
split makes it feel more natural and will make developers feel more
confident.
SIG-centric
---
* SIGs will form "bubbles" within this ring, possibly varying from packaging
guidelines in order to meet needs.
* Change management will also be SIG-focused.
Fedora already largely works this way. We just need more of it. If we can be
more flexible about what it means to participate in Fedora, we can increase
the intersection with upstream communities.
Fedora Commons
---
* The collection of packages outside Core following traditional policies
and practices
* We're not throwing that away
* Many people continue to be interested in this model
* Shared by other parts of Ring 2 where possible
This is a special bubble within Ring 2. In fact, on Day 0, it still is _all
of_ Ring 2. The name here is mean to evoke Creative Commons; we can discuss
other possibilities too, of course.
Many Ways!
---
* Software Collections
* Stacks 2.0
* OpenShift Cartridges
* Fedora Formulas
* Traditional Packaging
There are many different approaches people are working on right now. I don't
know which will ultimately be best. We want to enable the experimentation,
so we can find what succeeds. Some of these things aren't beautiful, but if
we can invite them in anyway, they can iterate towards perfection.
Exploring Different Approaches
---
* Packaged vs. DVCS
* Builds on Core vs. Overrides Core
* Replaces vs. Coexists
* Appliance vs. Shared
It may be useful as we think about the different approaches to think about
the different philosophies they take. For example, there are two different
ways to "override" something. One is to actually replace it — for example,
your Fedora Formula may replace /usr/bin/python with Python 3. The other is
to install multiple versions in parallel — using multi-versioned RPMs, or
Software Collections for whole stacks. In the future, a namespace approach
could even make /usr/bin/python be Python 3 for some processes but not for
the core OS.
Additionally, there's a natural distinction between things which build on
Fedora Core and may override Fedora Commons and those things which may
override both. As we explore these different approaches, it may be useful to
be intentional about which approach each SIG is using.
Ring Three: Applications
---
* User-level installable
* Shift focus away from RPMs
* Linux Apps... or Git with OpenShift
And, finally, the outside ring. This is end-user software. Exactly how this
will work is kind of further off, but it's important to think about in the
complete picture. We want to make this a nicer experience for Fedora users,
and we want to make it easier for application developers to offer their
stuff to Fedora.
Part Three: Concrete Steps
---
* What do we do to make this happen?
* What can be done now?
* What needs to be developed further?
To make this not just all talk....
Three Next Things
---
* Draw the "base design" for Fedora Core
* Make space for exploring Ring 2 innovation
* Emphasize upstream relationships (example: RubyGems.org)
Much of this I hope to have further developed in the next few weeks before
Flock. I'll talk a little bit about these three now, though.
First, I'm working on creating an example spin which would show a basic
suggestion for Fedora Core, both in a default install and an
everything-installed (including build deps) configuration. Right now, if we
take @standard + @core, that gives 430 packages from 330 SRPMs... but
recursively following the build deps explodes that to 3915 packages from
1800 SRPMs. That's smaller than the 13.7K SRPMs in all of Fedora, but bigger
than would be ideal for the Core. And it's probably not really the package
set we want. So, I'd like to look at that further, and I'll post more about
that here as I do.
Second, we can make some changes to allow Software Collections to be built
for Fedora. There's more we can do with OpenShift. And we can build up the
Fedora Formulas idea. Basically, let's do all that.
And third, by increasing our engagement upstream, we can reduce our own
work. For example, right now RubyGems.org doesn't do any validation of
licenses, basic review for malware, or gem signing. If we knew that this
basic diligence was happening upstream, we could extend our circle of trust.
We've long had the mantra of "upstream! upstream! upstream!" for code and
patches — we can do the same thing for packaging, for the same reasons and
for similar benefits. (But to do that, we need to work with upstream
packaging formats rather than demanding RPM — because experience shows that
that doesn't work.)
Conclusion
---
* Refocus Core to provide a better platform for building on
* Make room for innovation at the "Ring 2" level
* Empower SIGs to create solutions that fit
* Won't break what we have
* And we can start right now
So there we have it. Comments and discussion, please!
