This is a draft of the proposal I'm presenting at Flock, "An Architecture for a More Agile Fedora" (<http://sched.co/19ugKGM>). It represents a big change to how we as the Fedora project put together our distribution. I've gone through several drafts of this and talked to a few people, and I'd really value all of your feedback to refine it further. Preliminary slides are available at <http://mattdm.org/fedora/next/>; I will update them as this discussion progresses. (So, page numbers might change.) What follows is an annotated text version of the presentation — the annotations basically being what I might say as each slide displays. If you have a short attention span, you might want flip through the slides first and come back for explanation where it's unclear. And the explanation here might be unclear too — please ask questions. An Architecture for a More Agile Fedora presented by Matthew Miller ( ← with input from many people) === Problem Statement (What's this all about?) --- Let's Start with What's Good --- * Fedora is awesome * Great tech, great people * Amazing, high standards * Best enthusiast distro * Solid base for RHEL * Pretty decent cloud image I've been involved to some small degree with Fedora since the beginning. This is a project and a Linux distribution which I love, and which I know everyone reading this also cares about deeply. We don't want to break what is successful and works. But... --- * We need to be more than that * Not widely used by RHEL users * Not so great for building on... * Including Red Hat's own stuff * We're not seen as relevant... * Let alone exciting Whenever I go to a tech meetup or talk to someone from a new startup company, their developers are inevitably using a different (usually proprietary) desktop OS, plus a non-Fedora distribution on their code. We're being left behind and left out. It doesn't matter how theoretically great we are if we end up with no users. The Idea --- Focus core distro as platform, include layers of modular enabling technologies, and provide room for special interest groups to create solutions within the Fedora circle. Starting with Ring One --- * We draw a line around a small "base design" * E.g.: Everything in @standard, plus the toolchains to build it * Line could go elsewhere; that's an implementation detail * Most strict change management * Packaging standards as current Fedora The example line is somewhat arbitrary here — I'll talk a little more about practicalities at the end of the talk. The idea of using @standard as a starting place is just that. That's the group that's one step up from a minimal install; common things usually installed. That collection of packages is arbitrary and has grown arbitrarily, but it makes a place to begin. We'll probably end up with a set of packages that isn't exactly tied to any existing group. Anyway, this is the core OS. It's bigger than a minimal install, and is a complete but lightweight Linux distribution. It has a very clear charter: use the latest in open source platform-level software integrated into a solid foundation for the higher rings. The Base OS --- * Ring One is base functionality and behavior that everyone can expect of any Fedora system * Making a distinction helps focus engineering resources * And it makes a foundation that higher levels know they can trust Ring 1 : A Picture --- ********* ** ** ** ** * Ring 1 * * * * Fedora Core * ** ** ** ** ********* Wait, did you just say Fedora _Core_? Yes I did. But, this is not a return to the old Core + Extras, because the line is drawn based on _what_ and _how_ rather than on who works for what company. There were two big problems with the long-ago approach. First, the "core" was developed internally and not as a community open source project. If a package was in that part of the distribution, there was no real route for participation. Bad. Second, the quality standards for Fedora Extras, the collection of packages built around the core, were much, much higher. Upside-down! So, we won't repeat those mistakes. The new "Fedora Core" is a community project just like Fedora today. And we focus the most strict standards on the inner rings. Ring Zero: Fedora Minimal --- * Start from the current @core, and aim to shrink * Intentional focus on minimalism * JEOS — Just Enough Operating System * Even stronger rules than Ring 1 * Not self-hosting This is pretty self-explanatory. There is a lot of demand and many use cases for a "Just Enough Fedora" operating system. Ring 0 exists to provide that in a serious way which Fedora has never really covered before. Rings 1 + 0: Together --- (There is a diagram here, but I'm leaving it out of the text. It's a circle. With another circle.) Part Two... First, _Why?_ --- * Back to the problem statement: we need to be useful * People building on Fedora need a reliable base... * ... but need more flexibility in the areas relevant to their code * So let's design something that gives that ***DON'T PANIC*** This is a vision for the future. The Fedora Core idea can get started now, and the ideas beyond that are for development over a longer term. That doesn't mean never, and there are some specific things at the higher level to get started on now, but I'm not suggesting to change everything all crazily. We're not throwing out what have. The idea is to explore several different approaches which will better position Fedora as the distro of choice for future development. So... Rings 2 + 1 + 0 --- (There is another diagram here. It's more circles inside of circles.) Ring Two: Environments and Stacks --- * Environments are where you run the code you care about * Stacks are modular collections of _software which is used by other software_ In one of the earlier drafts, I had these as individual, separate rings. That turns out to be kind of distracting, because depending on your point of view either could easily be seen as inside or outside the other. So, I put them together in this level. Environments --- * Desktop environments, ideally with app containers * Platform as a Service, decoupled: integrated OpenShift Gears * Core Image (oVirt node, cloud guest) Stacks --- * Languages, databases, libraries * Maybe X/Wayland * The kind of things which could be OpenShift Cartridges or Software Collections (For Example) --- And there's another diagram here. In the Ring 2 circle, we have examples like Perl; Ruby 1.8.x, 1.9.x, 2.0; Python 2 & 3; Java, Wildfly; some databases, desktop environments, OpenShift, and then something called Fedora Commons, which I'll talk about in a bit. What's a Ring? --- * Rings give SIGs a way to replace the default expectations with their own * Separate policies may apply to each ring * But also, also, we need community, policies, and infrastructure to make them _part of Fedora_ So, there's nothing really magical about the circles. The main idea is that we need to make room for more exploration within Fedora beyond the traditional idea of how a Linux distribution is put together, and this is a model for describing how and where that can work. We still need global Fedora policies, but they'll apply differently to each ring. In Ring 2, SIGs will be given a lot of room to develop their own rules, but in Ring 1, not so much. Different environments and stacks within Ring 2 may have different individual policies from each other, but there will also be general policies for Ring 2 which are different from those for Ring 1, and of course global policies that apply to the whole thing (e.g., no proprietary code). Possible Examples --- * Possible example: bundled libs okay, but must be documented * Can override versions of software at lower tier * Can overlap with software from other stacks * Not necessarily even RPM * Different lifecycle (but shared release cadence) Big stuff here. Obviously, no-bundled-libs is a crucial part of the packaging guidelines today. As a sysadmin, I know why it's important. This is not just a noble goal, but also something that pragmatically makes systems better. But, it's also keeping us from having software that people really use in Fedora. Chef and Hadoop are two big examples. This hurts us more than it helps the world. So, in some areas, we need a different approach. Overriding other parts of the distribution is crucial, and I'll talk a little bit more about different approaches and what they mean in a little bit. Some approaches may even move beyond RPM packaging. This can mean focusing on "language native" package formats like Jars, Gems, and Eggs; or it can mean moving to a Git-based distribution model for some parts of the distro. And, different lifecycle: a SIG may decide that Ruby 1.9.x support should continue for some number of years and maintain that stack against different Fedora Core releases. We do this already in some ways, but having a clear split makes it feel more natural and will make developers feel more confident. SIG-centric --- * SIGs will form "bubbles" within this ring, possibly varying from packaging guidelines in order to meet needs. * Change management will also be SIG-focused. Fedora already largely works this way. We just need more of it. If we can be more flexible about what it means to participate in Fedora, we can increase the intersection with upstream communities. Fedora Commons --- * The collection of packages outside Core following traditional policies and practices * We're not throwing that away * Many people continue to be interested in this model * Shared by other parts of Ring 2 where possible This is a special bubble within Ring 2. In fact, on Day 0, it still is _all of_ Ring 2. The name here is mean to evoke Creative Commons; we can discuss other possibilities too, of course. Many Ways! --- * Software Collections * Stacks 2.0 * OpenShift Cartridges * Fedora Formulas * Traditional Packaging There are many different approaches people are working on right now. I don't know which will ultimately be best. We want to enable the experimentation, so we can find what succeeds. Some of these things aren't beautiful, but if we can invite them in anyway, they can iterate towards perfection. Exploring Different Approaches --- * Packaged vs. DVCS * Builds on Core vs. Overrides Core * Replaces vs. Coexists * Appliance vs. Shared It may be useful as we think about the different approaches to think about the different philosophies they take. For example, there are two different ways to "override" something. One is to actually replace it — for example, your Fedora Formula may replace /usr/bin/python with Python 3. The other is to install multiple versions in parallel — using multi-versioned RPMs, or Software Collections for whole stacks. In the future, a namespace approach could even make /usr/bin/python be Python 3 for some processes but not for the core OS. Additionally, there's a natural distinction between things which build on Fedora Core and may override Fedora Commons and those things which may override both. As we explore these different approaches, it may be useful to be intentional about which approach each SIG is using. Ring Three: Applications --- * User-level installable * Shift focus away from RPMs * Linux Apps... or Git with OpenShift And, finally, the outside ring. This is end-user software. Exactly how this will work is kind of further off, but it's important to think about in the complete picture. We want to make this a nicer experience for Fedora users, and we want to make it easier for application developers to offer their stuff to Fedora. Part Three: Concrete Steps --- * What do we do to make this happen? * What can be done now? * What needs to be developed further? To make this not just all talk.... Three Next Things --- * Draw the "base design" for Fedora Core * Make space for exploring Ring 2 innovation * Emphasize upstream relationships (example: RubyGems.org) Much of this I hope to have further developed in the next few weeks before Flock. I'll talk a little bit about these three now, though. First, I'm working on creating an example spin which would show a basic suggestion for Fedora Core, both in a default install and an everything-installed (including build deps) configuration. Right now, if we take @standard + @core, that gives 430 packages from 330 SRPMs... but recursively following the build deps explodes that to 3915 packages from 1800 SRPMs. That's smaller than the 13.7K SRPMs in all of Fedora, but bigger than would be ideal for the Core. And it's probably not really the package set we want. So, I'd like to look at that further, and I'll post more about that here as I do. Second, we can make some changes to allow Software Collections to be built for Fedora. There's more we can do with OpenShift. And we can build up the Fedora Formulas idea. Basically, let's do all that. And third, by increasing our engagement upstream, we can reduce our own work. For example, right now RubyGems.org doesn't do any validation of licenses, basic review for malware, or gem signing. If we knew that this basic diligence was happening upstream, we could extend our circle of trust. We've long had the mantra of "upstream! upstream! upstream!" for code and patches — we can do the same thing for packaging, for the same reasons and for similar benefits. (But to do that, we need to work with upstream packaging formats rather than demanding RPM — because experience shows that that doesn't work.) Conclusion --- * Refocus Core to provide a better platform for building on * Make room for innovation at the "Ring 2" level * Empower SIGs to create solutions that fit * Won't break what we have * And we can start right now So there we have it. Comments and discussion, please! -- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm@xxxxxxxxxxxxxxxxx> -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
