Re: Feedback sought: can we drop cgroupv1 support soon?

[Stéphane Graber <stgraber@xxxxxxxxxx>]

Hey there,

I believe Christian may have relayed some of this already but on my
side, as much as I can sympathize with the annoyance of having to
support both cgroup1 and cgroup2 side by side, I feel that we're sadly
nowhere near the cut off point.

>From what I can gather from various stats we have, over 90% of LXD
users are still on distributions relying on CGroup1.
That's because most of them are using LTS releases of server
distributions and those only somewhat recently made the jump to
cgroup2:
 - RHEL 9 in May 2022
 - Ubuntu 22.04 LTS in April 2022
 - Debian 11 in August 2021

OpenSUSE is still on cgroup1 by default in 15.4 for some reason.
All this is also excluding our two largest users, Chromebooks and QNAP
NASes, neither of them made the switch yet.

I honestly wouldn't be holding deprecating cgroup1 on waiting for
those few to wake up and transition.
Both ChromeOS and QNAP can very quickly roll it out to all their users
should they want to.
It's a bit trickier for OpenSUSE as it's used as the basis for SLES
and so those enterprise users are unlikely to see cgroup2 any time
soon.

Now all of this is a problem because:
 - Our users are slow to upgrade. It's common for them to skip an
entire LTS release and those that upgrade every time will usually wait
6 months to a year prior to upgrading to a new release.
 - This deprecation would prevent users of anything but the most
recent release from running any newer containers. As it's common to
switch to newer containers before upgrading the host, this would cause
some issues.
 - Unfortunately the reverse is a problem too. RHEL 7 and derivatives
are still very common as a container workload, as is Ubuntu 16.04 LTS.
Unfortunately those releases ship with a systemd version that does not
boot under cgroup2.

That last issue has been biting us a bit recently but it's something
that one can currently workaround by forcing systemd back into hybrid
mode on the host.
With the deprecation of cgroup1, this won't be possible anymore. You
simply won't be able to have both CentOS7 and Fedora XYZ running in
containers on the same system as one will only work on cgroup1 and the
other only on cgroup2.

Now this doesn't bother me at all for anything that's end of life, but
RHEL 7 is only reaching EOL in June 2024 and while Ubuntu 16.04 is
officially EOL, Canonical provides extended support (ESM) on it until
April 2026.


So given all that, my 2 cents would be that ideally systemd should
keep supporting cgroup1 until June 2024 or shortly before that given
the usual leg between releasing systemd and it being adopted by Linux
distros. This would allow for most distros to have made it through two
long term releases shipping with cgroup2, making sure the vast
majority of users will finally be on cgroup2 and will also allow for
those cgroup1-only workloads to have gone away.

I guess that would mean holding on to cgroup1 support until EOY 2023
or thereabout?

Stéphane

On Thu, Jul 21, 2022 at 5:55 AM Christian Brauner <brauner@xxxxxxxxxx> wrote:
>
> [Cc Stéphane and Serge]
>
> On Thu, Jul 21, 2022 at 11:03:49AM +0200, Lennart Poettering wrote:
> > Heya!
> >
> > It's currently a terrible mess having to support both cgroupsv1 and
> > cgroupsv2 in our codebase.
> >
> > cgroupsv2 first entered the kernel in 2014, i.e. *eight* years ago
> > (kernel 3.16). We soon intend to raise the baseline for systemd to
> > kernel 4.3 (because we want to be able to rely on the existance of
> > ambient capabilities), but that also means, that all kernels we intend
> > to support have a well-enough working cgroupv2 implementation.
> >
> > hence, i'd love to drop the cgroupv1 support from our tree entirely,
> > and simplify and modernize our codebase to go cgroupv2-only. Before we
> > do that I'd like to seek feedback on this though, given this is not
> > purely a thing between the kernel and systemd — this does leak into
> > some userspace, that operates on cgroups directly.
> >
> > Specifically, legacy container infra (i.e. docker/moby) for the
> > longest time was cgroupsv1-only. But as I understand it has since been
> > updated, to cgroupsv2 too.
> >
> > Hence my question: is there a strong community of people who insist on
> > using newest systemd while using legacy container infra? Anyone else
> > has a good reason to stick with cgroupsv1 but really wants newest
> > systemd?
> >
> > The time where we'll drop cgroupv1 support *will* come eventually
> > either way, but what's still up for discussion is to determine
> > precisely when. hence, please let us know!
>
> In general, I wouldn't mind dropping cgroup1 support in the future.
>
> The only thing I immediately kept thinking about is what happens to
> workloads that have a v1 cgroup layout on the host possibly with an
> older systemd running container workloads using a newer distro with a
> systemd version without cgroup1 support.
>
> Think Ubuntu 18.04 host running a really new Ubuntu LTS that has a
> version of systemd with cgroup1 support already dropped. People do
> actually do stuff like that. Stéphane and Serge might know more about
> actual use-cases in that area.
>
> But fwiw, we did have people show up with this and related problems for
> the last 5 years or so at conferences.
>
> Christian