A goal of rpm-ostree is to *empower* users and developers. Being able to fearlessly have automatic updates on and know that if your system loses power or the kernel freezes in the middle everything will be OK - that's empowering! But some implementation choices led to other restrictions that weren't intentional, just bugs. In this release we've made a large advance in lifting one of the biggest restrictions. ## rpm-ostree ex apply-live In this release the functionality formerly known as `rpm-ostree ex livefs` is now known as `rpm-ostree ex apply-live`, and it's been placed on a much firmer technical foundation and is considered very safe to use. It's still under `ex` because we may make some interface changes, and we hope to gather feedback. One can often see in discussions people say something of the form "with rpm-ostree you need to reboot for updates" (notice this is a phrase couched in terms of restrictions). But that's never been true in general, because the goal is to move most software to containers. Rather than having e.g. `gcc` as part of your root filesystem (in ostree), you have it in your development container updated independently from the host. Server and desktop applications run as containers, etc. For all the software that *is* part of the host though, our story until recently has been incomplete. We've had `rpm-ostree usroverlay` for a long time, which can be very convenient for testing things. But a core problem is that rpm-ostree has no idea what's happening in the transient writable layer, and further we didn't offer any tools to make changes there. The writable layer also meant that any modifications was allowed, even if all one wanted was e.g. installing a single package. For example, if you use `rpm-ostree usroverlay` and then `rpm -Uvh https://example.com/some.rpm`, `rpm-ostree status` doesn't show it as layered - it won't persist across upgrades and reboots if you did want that. Now you can combine e.g. ``` $ rpm-ostree install fish $ rpm-ostree ex apply-live ``` And have the newly layered `fish` package appear both for the *next* boot (persistently) as well as your running filesystem. An obvious thing to add will be `rpm-ostree install --apply-live fish` once we stabilize the interface (which will likely be soon). ### Implementation The big change in implementation that makes this very safe is that changes to the running filesystem tree go into an overlayfs with the upper directory being temporary. We no longer create a rollback deployment; rebooting will return the previously-booted deployment to its original state. More information in [this ostree PR](https://github.com/ostreedev/ostree/pull/2103) and an example below. ### Still readonly One thing that *hasn't* changed (compared to previous `ex livefs`) is that `/usr` is still mounted read-only from the perspective of the rest of the system. It's worth emphasizing this! In this model, rpm-ostree is still in full control, `rpm-ostree status` is showing you truth, etc. This is a good example how use of the term "immutable" is suboptimal when discussing an rpm-ostree based system. See [this blog entry](https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-hysteresis/) for more discussion on that. ### Testing fixes However this isn't just for layered packages; let's take the example of something commonly shipped in a base image like [NetworkManager](https://gitlab.freedesktop.org/NetworkManager/NetworkManager), [podman](https://podman.io/), or [OpenSSH](https://www.openssh.com/). Say you're running Fedora CoreOS stable, and you want to test a fix for podman that's in [the testing stream](https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/). After you've run e.g. `rpm-ostree rebase fedora/x86_64/coreos/testing` you can now confidently `rpm-ostree ex apply-live` to switch your running filesystem to that without rebooting and start running `podman` commands. Then you can report success/failure to an upstream issue or bug. ## override replace https://bodhi/... Another notable feature in this release is support for directly pulling builds from current Fedora testing/build tools [Bodhi](https://bodhi.fedoraproject.org/) and [Koji](https://koji.fedoraproject.org/koji/). A longstanding tension that rpm-ostree has created is that we want to test the *whole* build (ostree commit) that is shipped to users, but Bodhi is oriented around per-package changes. Switching to a `testing` stream pulls in everything. That's how it's shipped to users so we need to test it, but it also means you're testing everything at once. Now with rpm-ostree v2021.1 you can run e.g.: ``` $ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2020-2908628031 ``` to directly pull in a single Bodhi update applied relative to your booted system (presumably a `stable` stream) without changing anything else. You can also pull in Koji builds (that may or may not be in a Bodhi update): ``` $ rpm-ostree override replace https://koji.fedoraproject.org/koji/buildinfo?buildID=1625029 ``` This allows you to more directly interact with Bodhi's current model of testing individual updates. Note that like all usage of `override replace`, these versions are "pinned" until explicitly removed with a variant of `rpm-ostree override reset`. ## Combining features And of course, the above two headlining features combine! Let's say that you're a Fedora Silverblue user, and you're hitting a WiFi issue that is claimed to be fixed in an updated NetworkManager. With a combination like: ``` $ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-<updateid> $ rpm-ostree ex apply-live $ systemctl restart NetworkManager ``` You can quickly test out that change. What's powerful about using rpm-ostree for this is that if e.g. something goes wrong and the updated NetworkManager is crashing or nonfunctional, you still have the base version from the booted deployment! There's no need to carefully save previous versions or keep a recovery USB key on hand. To undo the above: ``` $ rpm-ostree override reset -a # note actually resets all overrides $ rpm-ostree ex apply-live $ systemctl restart NetworkManager ``` But let's look at a more problematic scenario: starting the updated NetworkManager triggers a latent bug in your specific laptop's WiFi card driver, causing a kernel panic and system lockup. In this scenario, *all you need to do is reboot* (e.g. hold the power button for 5 seconds on most PCs) and you will be back exactly to the previous deployment with the working kernel+NetworkManager combination. You don't even need to manually stop at the bootloader menu to choose a previous deployment! And a similar thing applies if the system actually crashes *while* we're mutating your running filesystem with `apply-live` - just reboot and everything will be fine. The reason for this is twofold. First, by default ostree "finalizes" deployments (adds a bootloader entry) only when the system is cleanly shut down, so we're not booted into the new root. Second, the overlayfs approach mentioned above means the "live applied" changes are written to a transient overlay rather than changing the persistent filesystem (underlying deployment). ## Other changes ## Countme support rpm-ostree now ships an `rpm-ostree-countme.timer` unit which fulfills the same purpose as the [DNF countme feature](https://fedoraproject.org/wiki/Changes/DNF_Better_Counting). This will allow Fedora counting of installed systems to include rpm-ostree-based variants like Fedora CoreOS, Fedora Silverblue, and Fedora IoT, while still preserving user privacy. Note the unit is currently *not* enabled by default. For more information, including how to disable it, see [the docs](https://coreos.github.io/rpm-ostree/countme/). ### Default to sqlite in F34+ Fedora 34 rpm-ostree based systems will switch to the sqlite backend by default, silencing a warning about use of the bdb backend. ### Shared library "hollowed out" Quite a while ago rpm-ostree started shipping a shared library `librpmostree-1.so.1.0.0`; mainly it's used by e.g. gnome-software to display package diffs between ostree commits. However this created a problem later because we duplicated a lot of code between our binary and the shared library, and that in turn caused unnecessary bloat. The build system and implementation of the shared library was reworked to fork off the binary and parse its output. This reduced the size (debug symbols stripped) from ~2.6MB to 320k (-~88%). ### Internals: FFI and Rust Large portions of rpm-ostree have already been ported to the [Rust programming language](https://www.rust-lang.org/), in order to improve the overall level of safety and resilience of the project. To further accelerate this transition, since the last release we are using [cxx.rs](https://cxx.rs/) as a code generation helper, which has the compelling feature of supporting fully safe bidirectional calls between C++ and Rust. This required switching the C code to "C that builds in C++ mode". For the sake of clarity though, the goal here is to accelerate our transition to Rust, not to rewrite rpm-ostree code in modern C++. # In case you missed it: Some prior release highlights [v2020.6](https://github.com/coreos/rpm-ostree/releases/tag/v2020.6) included an `ex initramfs-etc` command to customize the initramfs and rewrote the documentation. [v2020.5](https://github.com/coreos/rpm-ostree/releases/tag/v2020.5) made a notable improvement in layered package handling, and also had some smaller milestones in porting some C code to Rust. [v2020.3](https://github.com/coreos/rpm-ostree/releases/tag/v2020.3) deprecated the non-`--unified-core` code - if you're still seeing this warning, please adjust your buildsystem! [v2020.2](https://github.com/coreos/rpm-ostree/releases/tag/v2020.2) contained a mix of bugfixes and features, but also continued a big push for improved continuous integration. [v2020.1](https://github.com/coreos/rpm-ostree/releases/tag/v2020.1) added several enhancements on the build side, and kicked off improved CI. # Bodhi update Here's the [rpm-ostree v2021.1 Bodhi update](https://bodhi.fedoraproject.org/updates/FEDORA-2021-e55da2fc78). We hope you give it a try (particularly the new `ex apply-live`)! _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
