Immutable Images: Single Data Patition

[Adrian Vovk <adrianvovk@xxxxxxxxx>]

Hello everyone!

Quick background: I'm working on carbonOS, a Desktop Linux distro
that's implementing your "GnomeBook" OS concept. I've stumbled upon
some limitations and would like to discuss solutions w/ the community.
Please see my previous email for a little more context.

This email is going to propose a way to remove the need for multiple
data partitions

==============================================================

Part of the A/B approach involves two classes of user data partitions:
ones that are encrypted (/var, /etc) and ones that are not (/home).
I'll be adding an additional partition to the non-encrypted category,
as part of my efforts to let Flatpak share the most common runtimes
across installations [1]. Note that I'm only talking about the data
partitions; the rootfs and/or usr partitions will be fixed size (~3gb
each in my case, which should be ample headroom)

Here's the issue: as a distributor I really have no idea how much
space each of these partitions will need. /var can either be tiny
(just storing some logs, some cache, other things like that) or it can
be many GB (storing A/B versioned container images, sysexts, portable
services, a system-wide Flatpak installation, etc). My Flatpak shared
installations directory can be < 1GB (if the user uses few apps, or
only GNOME apps, for instance) or much larger (if the user has some
out-of-date apps, mixes GNOME/KDE apps, etc). The /home directory
should probably by the biggest it could possibly be to maximize space
for user files. I could pick some percentages, of course, but that
would be a compromise that I don't think is strictly necessary and
will leave most people unhappy.

Letting the user choose a size themselves is also impractical. First
of all, I don't think the user would necessarily know any better than
I would how big they need these partitions to be: I've personally been
bitten by running out of space in / with plenty of unused space in
/home on Fedora [3]. In the event that the user guesses the wrong
number, what then? Or maybe they guess the right number but then years
later their requirements change: they find a new container-intensive
hobby, or change employers, or any number of other things. I don't
think it's a good answer to tell them to just reinstall; that is very
inconvenient and not something they've ever had to do with the other
OSs. It's a difficult solution on a technical level too. If I prompt
the user at install-time, then devices with the OS pre-installed won't
have the prompt & nor would a factory reset. I can ask on first boot,
when the partition is being created, but then I'm going to have to do
that from the initramfs (CLI? Plymouth? Put a GNOME session into the
initrd?). Overall, I don't think this is a viable solution.

I think this can be solved by having one user state partition w/ an
encrypted loopback file in it. As far as I can tell this is the
solution taken by ChromeOS [2]. There could then be a userspace
service that dynamically resizes the file up and down as-needed and as
space allows. I propose that we implement something similar in
systemd. Here's what I think is necessary:

- Assign a new partition type UUID. I don't think it needs to be done per-arch
- Have gpt-auto-generator automatically mount partitions w/ this UUID
into /state
- Once /state is mounted, have gpt-auto-generator automatically mount
/state/encrypted.img onto /state/encrypted (decrypting it via TPM)
- Expand systemd-repart to handle creating this /state partition and
the /state/encrypted.img blob inside of it (a unit that runs
systemd-repart to create the encrypted.img blob could probably
suffice? Or mkfs/repart/whatever automatically creates
/state/encrypted.img as-needed so that it works with --root or
--image).
- Implement a userspace service that grows/shrinks the
/state/encrypted.img file as needed. We could probably reuse parts of
homework here??

Then we could also go a step further to define a standard layout for
this /state partition. Could be as simple as defining some
bind-mounts: /state/home -> /home, /state/encrypted/var -> /var,
/state/encrypted/etc -> /etc (see section on /etc below),
/state/encrypted/usrlocal -> /usr/local, /state/encrypted/opt -> /opt,
/state/encrypted/srv -> /srv, etc. Perhaps these directories can have
un-encrypted variants for situations where people don't need them to
be encrypted. Maybe there could be a generator that mounts everything
in /state/encrypted and in /state to the appropriate destination based
on escaped filenames relative to / (e.g. /state/home -> /home,
/state/encrypted/my-fancy-dir -> /my/fancy/dir,
/state/encrypted/fancy\x2ddir -> /fancy-dir, etc).

Thoughts?

Thank you,
Adrian Vovk

[1]: https://github.com/flatpak/flatpak/issues/5254
[2]: https://www.chromium.org/chromium-os/chromiumos-design-docs/boot-design/#filesystem-initialization-and-layout
[3]: https://pagure.io/fedora-workstation/issue/152