Re: systemd-sysupdate support for slow rollout (aka A/B testing)

[Simon McVittie <smcv@xxxxxxxxxxxxx>]

On Tue, 02 Jan 2024 at 11:16:15 +0100, Lennart Poettering wrote:
> The idea so far was always that the server is dumb, and the client
> picks the release it wants.
> 
> I have thought about this usecase a while back, and my thinking was
> that such a staged update logic should be driven by the machine
> ID. i.e. we should teach sysupdate a simple logic that allows pattern
> matching of new versions based on some arithmetic of the machine
> ID. More specifically, include some value in the URL pattern that
> indicates the percentage of hosts that shall update to this
> release. Then, each client takes its machine ID, treats it as an
> integer and calculates modulo 100 of it or so, and then checks if the
> resulting value is below the intended percentage, and if so it
> updates, otherwise it doesn't.
> 
> (or something like that, the above is probably not ideal, since it
> would mean it's always the same hosts that try a new release first,
> and it probably should be evened out across the set of clients).

Prior art: Debian/Ubuntu apt does slow rollout for packages like
this, with simple filesystem-based http mirrors combined with "smart"
clients. It works by adding a Phased-Update-Percentage field to the
metadata of each package. The client calculates some sort of ID for itself
(I don't know precisely how), and then takes the upgrade if it finds that
its ID is in the first x% of the available range.

If I understand correctly, Ubuntu is using this mechanism in production
but Debian is not.

Using some sort of hash of the machine ID + the proposed version would
probably have the behaviour you want, of choosing a different x% of
machines to be the early-adopter set for each update?

> This would then mean for the server that it would first serve
> foobar_47.11_3.raw which would be version 47.11 of the OS, and 3% of
> the hosts would update to it. And then, once you collected enough
> feedback you'd rename the file to foobar_47.11_25.raw and 25% of the
> hosts would switch over. Finally you'd set the value to 100 (or maybe
> just drop it, which should be considered equivalent to 100), and then
> all remaining hosts would update.

If you're using a hash of the machine ID + the proposed version as
your randomization, then I think you'd want to have a single image (or
version ID, or some other unique identifier) for each proposed update, and
separately, a metadata field that sets *x* in the instruction "if you have
figured out that you are in the first x% of machines, upgrade". Otherwise,
publishing foobar_47.11_3.raw followed by foobar_47.11_25.raw would be
more likely to result in approximately (3% + 25% = 28%) of machines
upgrading[1], because the client doesn't know that it's actually the
same update and would "re-roll the dice" for each republished name.

    smcv

[1] more precisely, (0.03 + 0.25 - (0.03 * 0.25)) because of how
    conditional probabilities combine