I'm looking forward to the future of CentOS Stream

[Gordon Messmer <gordon.messmer@xxxxxxxxx>]

Personally, I think that changing focus on CentOS Stream is going to 
make CentOS (and maybe even RHEL) better in the same way and for the 
same reasons that Fedora is a better distribution than Red Hat Linux 
was.  CentOS Stream should fix the biggest problems that CentOS has had 
in the past, providing a reliable, free LTS distribution with community 
participation.

Having read the announcement, along with hundreds of reactions in blogs, 
forums, and mailing lists, I am of the opinion that all (or a reasonable 
approximation thereof) of the vocal concern is the result of the 
overloaded term "stable" as it applies to software distributions.  If we 
imagine a spectrum of individuals in which one end of the spectrum is 
individuals whose primary occupation is in release engineering for 
software distributions and the other end is individuals who primarily 
consume software distributions, I would expect that individuals on one 
end to mostly use the term "stable" in the sense of compatibility and 
semantic versioning, and the other end to use the same term in the sense 
of having fewer bugs.  The use of that word causes people at one end of 
the spectrum to infer a completely different message than people at the 
other end intend to communicate.

If we never use the ambiguous term "stable" and instead use the terms 
compatibility and reliability (the two common meanings of "stable" at 
different ends of the spectrum), I think that various aspects of CentOS 
Stream will be better than CentOS, or the same as CentOS.

With respect to compatibility:

I think most developers are familiar with semantic versioning.  Semantic 
versioning is used by some applications and libraries to indicate the 
nature and extent of changes.  The version is presented numerically as 
Major.Minor.Revision.  When new interfaces are added, the minor number 
is increased.  Those changes don’t affect backward compatibility.  When 
individual interfaces are changed or removed, the major number is 
increased.  Those changes aren’t backward compatible.  That allows 
consumers to infer that if an application is compatible with 8.1, then 
it is also compatible with 8.2 and later, but might not be compatible 
with 9.0.

Red Hat Enterprise Linux applies that concept to the entire software 
distribution, providing independent software vendors a convenient means 
of communicating their compatibility.  If they’ve tested their 
application on RHEL 8.2, then any RHEL 8 host patched to at least that 
release is expected to run the application.  Moreover, Red Hat will 
continue to publish security patches to each given minor release’s 
channel, allowing consumers to "pin" a host to a minor release.  Those 
hosts will not receive feature updates, but will mitigate vulnerabilities.

CentOS Stream isn’t going to feature minor releases, and isn’t going to 
provide semantic versioning of the distribution.  The same application 
that the vendor has validated on RHEL 8.2 will run on a fully patched 
CentOS Stream 8 host, but might not run on a host that isn’t fully 
patched.  On the surface, it appears that CentOS users will lose the 
convenience provided by semantic versioning.  I would simply point out 
that the CentOS developers have never supported running CentOS in any 
state other than fully patched.  They don’t publish security information 
in the package repositories, and they don’t support any means of pinning 
a host to a minor release.

For practical purposes, CentOS Stream will need to be fully patched for 
compatibility purposes, just like CentOS is, and will be equally suited 
for production purposes.

To put a really find point on that: Semantic versioning is only 
meaningful for hosts that are not fully patched.  A fully patched host 
is expected to be compatible with any application validated for that 
major release.

With respect to reliability:

Many of the people concerned about the change in focus refer to CentOS 
Stream as a "beta" for RHEL.  That is not how Red Hat or the CentOS 
maintainers describe CentOS Stream( anywhere that I've seen), and I 
think it ignores most of the development, testing, and distribution 
pipeline.

At the risk of oversimplifying that pipeline a whole lot, in the future 
Free Software will pass through several stages on the way to RHEL:

    Stage 1: (Software Development) The majority of development and 
testing is done in individual upstream projects, outside of Red Hat.

    Stage 2: (Release Development, aka Rawhide) The initial work to 
build and integrate individual packages with the rest of the software 
distribution is done in what is essentially a development branch of the 
software distribution.

    Stage 3: (Stable[1], aka Fedora) Packages that have passed through 
review and QA are published for general use.  There is no minor release, 
as major releases occur every 6 months and are supported for only 13 
months, anyway.  Compatibility is maintained by prohibiting significant 
version changes for applications and libraries "whenever possible." 
Users expect no new features during a release, but no breaking changes 
either.

    Stage 4: (Long Term Support, aka CentOS Stream)  Packages that 
CentOS Stream includes from Fedora have proven reliable in a variety of 
workloads managed by many users of Fedora.  These packages are expected 
to be very reliable as a result of testing by their developers, by 
package maintainers, and by real-world users.  They are included in 
CentOS Stream when they are ready.

    Stage 5: (Long Term Support with Semantic Versioning of the OS, aka 
Red Hat Enterprise Linux)  Packages that RHEL includes from CentOS 
Stream have a similar level of QA, but package updates that introduce 
new features and interfaces are accepted only once every six months when 
Red Hat publishes a minor release.

There’s a lot of concern that CentOS Stream will offer less reliability 
than RHEL, but there’s currently no reason to believe that will be true. 
 There is no evidence that the minor releases that are part of the RHEL 
lifecycle improve reliability, and as far as I know that's not the 
reason they're used.  Their function is to offer semantic versioning of 
the OS.

CentOS Stream will probably offer the same level of compatibility and 
reliability that CentOS does today, and should be equally appropriate or 
inappropriate for production use in the future as CentOS is now in that 
respect.  And that brings me to the one aspect where I think CentOS 
Stream will resolve the one major, glaring problem that CentOS has 
today, that most users ignore: Security.

With respect to security:

Today, CentOS is a release stage after Stage 5 described above.  The 
CentOS maintainers begin work on a minor release after that release is 
available to RHEL consumers, and the process of rebuilding those 
packages is often very time consuming.  CentOS maintainers have to 
reverse-engineer the exact order in which packages are built, with the 
exact set of installed and available packages in the build environment 
in order to ensure that the resulting package actually uses the same 
interfaces that RHEL’s packages do.  All packages require that ordering 
and build environment matching, but most packages are published in small 
sets and ordering is much easier to identify than it is when they are 
published in a large batch.

As a result, security updates can’t be published for CentOS while the 
maintainers are rebuilding the minor release, because the build 
dependencies aren’t available yet.  Those windows occur every six 
months, and are typically a month or more in length. [2]

Today, CentOS users accept the risk that for roughly two months out of 
the year, their systems may have known vulnerabilities with no patch to 
remediate the problem.  Personally, I think that’s a huge risk that 
needs to be weighed against the costs of RHEL licenses whenever CentOS 
is used in production.

The good news is that CentOS Stream looks like it won't have that 
problem.  CentOS Stream updates still won’t be prepared early, while 
vulnerability details are embargoed, but there aren’t any windows in 
which CentOS Stream can’t immediately begin work on preparing updates 
once the embargo ends.  That means that CentOS Stream will be as secure 
as CentOS is today in between minor updates, and significantly more 
secure than CentOS is today while its maintainers prepare minor releases.

The trade-off of significantly improving security in exchange for giving 
up semantic versioning of the OS is a huge improvement for production 
purposes.

In addition, the announcement of the change in focus indicates that 
Fedora maintainers should have much better visibility into work going on 
within the RHEL release engineering process, and more opportunity to 
participate in that work, and I look forward to that.  CentOS’s big 
security problem has always been compounded by the lack of any external 
visibility into the process.

Based on the information available today, I expect CentOS to be a very 
reliable, reasonably secure distribution of GNU/Linux with Long Term 
Support.  And judging by Red Hat’s mention that Facebook’s internal 
groups either are already using an internally curated OS built from 
CentOS Stream, or will be using it soon, I think I’m not alone in 
believing that.


1: I’m using the term "stable" here because I expect it to have both 
common meanings: compatibility isn’t broken within a release, and the 
software is expected to be reliable.

2: https://en.wikipedia.org/wiki/CentOS#CentOS_version_8

_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
https://lists.centos.org/mailman/listinfo/centos