Re: Immutable Images: Updating

[Lennart Poettering <lennart@xxxxxxxxxxxxxx>]

On Di, 21.02.23 16:42, Adrian Vovk (adrianvovk@xxxxxxxxx) wrote:

> For OTA updates, I have some design goals I want to achieve, and I
> think it's doable with systemd-sysupdate if I extend it in some places
> and work around it in others. First, however, here are some of the
> features I'd like that I don't think sysupdate does natively (I'll
> follow with some concrete proposals for implementation):
>
> - Optional "delta" updates: carbonOS is working on chunk-based updates
> for squashfs, which I call delta updates for simplicity. We parse the
> squashfs file to get a list of compressed chunks, hash each chunk, and
> then store the hash+location+size in a database. Then the "patch" is
> computed from this database and basically just looks like this: "copy
> byes x thru y out of the old version, then w thru z out of the new
> version, then ....". These deltas are optional, so if they're missing
> we fall back to a full download of the image. This is similar to
> casync in idea, except it doesn't need any special layout on the
> server and creates its chunks w/ awareness of the input file format.
> The only requirement on the server side is support for http range
> requests (and to serve the delta "patches", of course!)

I sympathize. My intention for systemd-sysupdate is that the backends
eventually become pluggable. i.e. we just ship a default boring http
backend, but you could drop other backends in and then use them
directly. One of these backends could be casync, but of course
anything else is fine too.

systemd-sysupdates would then provide via some env var paths to the
older version(s) of the resource to update, which the backends can use
to speed up things in a delta fashion.

I think making this pluggable wouldn't be that hard: the download
components are out of process anyway, and shared with importd. It
would just be a matter of slightly cleaning this up, and then somehow
deriving the right plugin to use frm the url scheme. i.e. my thinking
was that if you pass a URL such as "https+casync://foobar/…" this
would result in a backend binary
/usr/lib/systemd/download/https+casync to be invoked rather than
/usr/lib/systemd/systemd-import like now. People would then symlink
those paths to their binary of choice for the url schemes of their
choice.

I figure this would be a 20 line patch. Would be happy to review a
patch for that.

> - Peer updates: For situations where users have >1 machine running
> carbonOS on the same architecture, it'd be nice if these machines can
> share update payloads instead of fetching them from the internet. When
> trying to fetch some resource from the update server, instead we first
> look for peers that might have this resource. This can work
> transparently with deltas (i.e. the server running on the peer will
> support range requests). If a user has a desktop and a laptop both
> running carbonOS, and the desktop updates overnight, it will have the
> full update payload located in one of its A/B partitions that it can
> then serve to the laptop whenever the laptop is awakened and can
> update.

Yeah, this is on the TODO lits for casync too.

I sympathize with the goal, but I think this probably should be hidden
in the backend, and not leak into systemd-sysupdate.

> - Update suspend/resume: Basically I'd expect that if an update gets
> interrupted it can be seamlessly resumed at a later point in time
> without having to restart all the progress that has been done
> previously. If a laptop is 50% of the way through downloading a 2 GB
> image when the user closes the lid and moves locations, I don't see a
> reason to re-download the first GB of content when the machine comes
> back online.

In theory a good chunking downlaod should be able to detect that
automatically.

But yeah, you are right.

> - DBus Service: This will allow sysupdate to be used w/o root access
> via polkit. I'd be able to write a gnome-software plugin to integrate
> w/ systemd-sysupdate, which will allow people to update their system
> via the GUI. Sysupdate w/o dbus is great for a use-case where it is
> triggered from a systemd service, but a general purpose desktop OS
> will need to report to the user via the GUI.

Sure, makes sense.

> - Changelogs: Basically a list of changes that happened between two
> versions of some resource that can be concatenated together and
> presented to the user. Also should probably be translated. So, if the
> user is updating from version A to C (and skipping B), you fetch the
> A->B changelog, then the B->C changelog, pick the right language,
> concatenate the changelogs together, and present it to the user. This
> needs to be done before the update is downloaded & applied, and it
> needs to be done per resource that is updated (though not every
> resource will have a changelog)

Hmm, so this is an "interactive" feature. So far that was out of scope
for systemd-sysupdate.

Should this really be done within systemd-update though? wouldn't it
make more sense, to allow declaration of a "ReleaseNotes=" link inside
a sysupdate .conf file, that can optionally take an URL parameter with
the old and the new version? Then, rather than actually doing much
client-side with this, we'd just show the formatted URL, and allow users
to click on it before they do there thing. i.e. the diffing of the
changelogs would be done server-side (if desired).

i.e. ReleaseNotes=https://myos.com/;oldversion=@x@;toversion=@y@

> - Sysexts: Sysexts should be updated in lock-step w/ the OS image.

well, i'd make that "soft" coupling, i.e. we should have a way how
systemd-sysupdate can go through all DDIs installed locally, i.e. the
OS itself, sysext, syscfg, portable services, containers, update them
to newest, and that's it. Enforcement what fits together would then be
up to systemd-sysext/systemd-syscfg and the other tools.

> Here are my ideas for implementation:
>
> - Deltas and peering: I think I can make this completely transparent
> to sysupdate. Basically I'll just have a web server running on
> localhost that exposes a directory structure of complete images (no
> deltas, no peering), but intercepts requests for these images and
> first tries to fetch a delta and/or peer the update. I'll call this
> the "proxy" server. Then I can just point sysupdate at this proxy
> server and it will have delta/peering features without needing to be
> aware of them whatsoever.

Sounds fine to me, as long as we agree this is outside of
systemd-sysupdate's scheme.

But as mentioned I'd not even bother with the proxying, but just do a
clean out-of-process plugin interface. We basically have that already
as mentioned as we call into systemd-import binary for the actual
downloads.

> - Update resumption: I think we can assume that the server does http
> range requests (and if not, we don't support resumption). When
> downloading to a file I think this is pretty trivial: just send a
> range-request to exclude the already-downloaded content and append to
> the file. When downloading to a partition, however, I'm unsure that
> it's quite so simple: I don't know how you could detect how far along
> into a download you were when interrupted. Perhaps just cache your
> progress somewhere in /var/tmp or similar?

I though about this before in context of casync. I think a simply idea
would be to just write a size value (or bitmap) to the end of the
partition you are about to write, and make sure you download/write the
last sector of what you write last, so that you can make use of that
info as long as possible,

i.e. a downloader that recognizes it is writing directly to a partition would
read a sector at the end of the partition, check if it carries some
recognizable signature, and if so use the information from it to
proceed with downloading only what's not downloaded yet. It would then
keep updating that sector with far it has progressed.

All this should be a hint only, i.e. it must verify hashes of what it
downloaded anyway, and when it finds parts have already been
downloaded it must verify they actually match expectations.

> - DBus: I can write a wrapper dbus service that just forks off the
> systemd-sysupdate binary, however that will make it difficult to
> justify a gnome-software plugin upstream (one of the prereqs is that
> the updater itself is something that will be in use & supported; I
> think being part of systemd meets that bar).

I don't grok this.

Preqreq for what is what?

> Also that would prevent users from updating from the command line
> w/o root, unless I implement my own wrapper CLI tool as well. I'd
> prefer to implement a dbus service for sysupdate upstream. Here's my
> thinking: we make an `updatectl` binary w/ systemd-sysupdate's
> current CLI and make systemd-sysupdate a dbus service (in
> /usr/lib/systemd). For any people already using the
> systemd-sysupdate command directly, a symlink
> /usr/bin/updatectl->/usr/bin/systemd-sysupdate can be installed
> (controlled via meson option) to give them time to transition, to be
> eventually dropped. This will make sysupdate consistent with the
> other systemd services

So to say this clearly, for me the primary focus was always automatic
updates, not user triggered ones. i.e. rather than have users manually
download a new release and upgrading to it, we'd always download it in
the bg when available and all conditions to do so are met, and then
just ask the user "hey, we updated, please reboot". This is what
chromeos and so on all do.

Out of this model you see the the permssion concepts we have in place,
and that we are root-only: it was never the intention for regular
users to directly issue updates.

That said, I sympathize with your use case. But even then I think a
D-Bus service that invokes systemd-sysupdate is actually totally the
way to go. i.e. systemd-sysupdate should not talk dbus natively,
because it is a synchronous tool, it does't have an async event
loop. It is actually already written in a style that it is tool that
can nicely be called from other subsystems. i.e. it can generate
output as JSON, and notifications via sd_notify(). Because noone is
using this yet, this might need more love to be complete (for example,
we should propagate the sd_notify() messages the systemd-import
backend generates, we currently don't afair), but conceptually the
basis is all there.

So yeah, a "systemd-sysupdated" that provides a dbus-interface (and/or
varlink) around "systemd-sysupdate" makes total sense to me. We
already have a similar service btw, i.e. "systemd-importd" calls out
to "systemd-import" to do the actual work. (And yes this might mean we
have a process tree where "systemd-sysupdated" has children
"systemd-sysupdate" that in turn have "systemd-import": the uppermost
process does dbus, the middle process does version mgmt, and the
innermost process does the actual downloading. And yes, it's entirely
fine).

> - Changelogs: I think this will just need to be implemented into
> systemd-sysupdate. I think one thing to simplify is to just remove the
> concatenation process - that can be done either on the server-side or
> as part of the proxy server. Then a [Source] could grow a
> "ChangelogPattern" option that identifies the changelog to fetch for a
> given transfer. You could have a @L wildcard for the user's language
> (obtained from the DBus API/systemd-sysupdate's environment).
> ChangelogPattern would take a list of patterns to handle changelogs
> when versions are skipped. So average usage would be:
> `ChangelogPattern=changelog-%w-@v-@xxxxx changelog-@v-@xxxxx
> changelog-@xxxxxxxx` or similar.

Wouldn't the URL thing suffice I suggested above?

I mean, people need networking anyway, and typically it should be much
nicer to reference a website with release notes than some text based diffing.

> - Sysexts: The sysext can drop a file into
> /usr/lib/sysupdate.d/sysext-id.conf to update itself. I suppose that
> means the sysext would need to be put in some well-known place, but
> for my purposes that's good enough. Perhaps down the road
> systemd-sysext can intelligently create sysupdate files based on a
> template shipped in the sysext, or sysupdate itself can look for
> updatable sysexts, but that's a different discussion for a different
> place I think. As far as I can tell this issue of updating sysexts is
> already on your radar.

systemd-sysupdate already has an "--image=" switch which allows
updating arbitrary DDIs if they carry sysupdate info.

So my idea was to eventually have "systemd-sysupdate --all" which
would iterate through all places we might have DDIs:

* /usr/lib/extensions/
* /var/lib/machines/
* /usr/lib/syscfg/
* /usr/lib/portables/
* the root block device itself

And then one-by-one update them as if you'd call systemd-sysupdate
individually on each via "--image=".

Lennart

--
Lennart Poettering, Berlin