Immutable Images: Updating

[Adrian Vovk <adrianvovk@xxxxxxxxx>]

Hello everyone!

I'm working on carbonOS, a Desktop Linux distro that will be
implementing many (most?) of the ideas proposed by systemd's
developers in the last while regarding UKIs, DDIs, sysext, "Fitting
everything together", etc. Basically I'm implementing the "GnomeBook"
OS you've been asking for in your blog posts. It's a fully independent
distro which gives me complete freedom from any packaging legacy,
compatibility, project politics, etc. This isn't a hypothetical
project: I have had a functioning desktop OS for a few years using
ostree, but I find the layout you propose much more compelling

As part of figuring out how to put together such a system in practice,
I've come up with a few questions and missing bits. So, I'd like to
discuss with the community here and see what kinds of solutions I can
help get upstream. I'm going to be sending more emails in the coming
hours/days/weeks regarding these questions.

This email is going to be talking about systemd-sysupdate

==============================================================

For OTA updates, I have some design goals I want to achieve, and I
think it's doable with systemd-sysupdate if I extend it in some places
and work around it in others. First, however, here are some of the
features I'd like that I don't think sysupdate does natively (I'll
follow with some concrete proposals for implementation):

- Optional "delta" updates: carbonOS is working on chunk-based updates
for squashfs, which I call delta updates for simplicity. We parse the
squashfs file to get a list of compressed chunks, hash each chunk, and
then store the hash+location+size in a database. Then the "patch" is
computed from this database and basically just looks like this: "copy
byes x thru y out of the old version, then w thru z out of the new
version, then ....". These deltas are optional, so if they're missing
we fall back to a full download of the image. This is similar to
casync in idea, except it doesn't need any special layout on the
server and creates its chunks w/ awareness of the input file format.
The only requirement on the server side is support for http range
requests (and to serve the delta "patches", of course!)

- Peer updates: For situations where users have >1 machine running
carbonOS on the same architecture, it'd be nice if these machines can
share update payloads instead of fetching them from the internet. When
trying to fetch some resource from the update server, instead we first
look for peers that might have this resource. This can work
transparently with deltas (i.e. the server running on the peer will
support range requests). If a user has a desktop and a laptop both
running carbonOS, and the desktop updates overnight, it will have the
full update payload located in one of its A/B partitions that it can
then serve to the laptop whenever the laptop is awakened and can
update.

- Update suspend/resume: Basically I'd expect that if an update gets
interrupted it can be seamlessly resumed at a later point in time
without having to restart all the progress that has been done
previously. If a laptop is 50% of the way through downloading a 2 GB
image when the user closes the lid and moves locations, I don't see a
reason to re-download the first GB of content when the machine comes
back online.

- DBus Service: This will allow sysupdate to be used w/o root access
via polkit. I'd be able to write a gnome-software plugin to integrate
w/ systemd-sysupdate, which will allow people to update their system
via the GUI. Sysupdate w/o dbus is great for a use-case where it is
triggered from a systemd service, but a general purpose desktop OS
will need to report to the user via the GUI.

- Changelogs: Basically a list of changes that happened between two
versions of some resource that can be concatenated together and
presented to the user. Also should probably be translated. So, if the
user is updating from version A to C (and skipping B), you fetch the
A->B changelog, then the B->C changelog, pick the right language,
concatenate the changelogs together, and present it to the user. This
needs to be done before the update is downloaded & applied, and it
needs to be done per resource that is updated (though not every
resource will have a changelog)

- Sysexts: Sysexts should be updated in lock-step w/ the OS image.

Here are my ideas for implementation:

- Deltas and peering: I think I can make this completely transparent
to sysupdate. Basically I'll just have a web server running on
localhost that exposes a directory structure of complete images (no
deltas, no peering), but intercepts requests for these images and
first tries to fetch a delta and/or peer the update. I'll call this
the "proxy" server. Then I can just point sysupdate at this proxy
server and it will have delta/peering features without needing to be
aware of them whatsoever.

- Update resumption: I think we can assume that the server does http
range requests (and if not, we don't support resumption). When
downloading to a file I think this is pretty trivial: just send a
range-request to exclude the already-downloaded content and append to
the file. When downloading to a partition, however, I'm unsure that
it's quite so simple: I don't know how you could detect how far along
into a download you were when interrupted. Perhaps just cache your
progress somewhere in /var/tmp or similar?

- DBus: I can write a wrapper dbus service that just forks off the
systemd-sysupdate binary, however that will make it difficult to
justify a gnome-software plugin upstream (one of the prereqs is that
the updater itself is something that will be in use & supported; I
think being part of systemd meets that bar). Also that would prevent
users from updating from the command line w/o root, unless I implement
my own wrapper CLI tool as well. I'd prefer to implement a dbus
service for sysupdate upstream. Here's my thinking: we make an
`updatectl` binary w/ systemd-sysupdate's current CLI and make
systemd-sysupdate a dbus service (in /usr/lib/systemd). For any people
already using the systemd-sysupdate command directly, a symlink
/usr/bin/updatectl->/usr/bin/systemd-sysupdate can be installed
(controlled via meson option) to give them time to transition, to be
eventually dropped. This will make sysupdate consistent with the other
systemd services

- Changelogs: I think this will just need to be implemented into
systemd-sysupdate. I think one thing to simplify is to just remove the
concatenation process - that can be done either on the server-side or
as part of the proxy server. Then a [Source] could grow a
"ChangelogPattern" option that identifies the changelog to fetch for a
given transfer. You could have a @L wildcard for the user's language
(obtained from the DBus API/systemd-sysupdate's environment).
ChangelogPattern would take a list of patterns to handle changelogs
when versions are skipped. So average usage would be:
`ChangelogPattern=changelog-%w-@v-@xxxxx changelog-@v-@xxxxx
changelog-@xxxxxxxx` or similar.

- Sysexts: The sysext can drop a file into
/usr/lib/sysupdate.d/sysext-id.conf to update itself. I suppose that
means the sysext would need to be put in some well-known place, but
for my purposes that's good enough. Perhaps down the road
systemd-sysext can intelligently create sysupdate files based on a
template shipped in the sysext, or sysupdate itself can look for
updatable sysexts, but that's a different discussion for a different
place I think. As far as I can tell this issue of updating sysexts is
already on your radar.

Thoughts?

Thank you,
Adrian Vovk