Re: soft-reboot and surviving it

[Luca Boccassi <luca.boccassi@xxxxxxxxx>]

On Fri, 19 Apr 2024 at 12:50, Thorsten Kukuk <kukuk@xxxxxxxx> wrote:
>
> On Fri, Apr 19, 2024 at 11:48 AM Luca Boccassi <luca.boccassi@xxxxxxxxx> wrote:
> > On Fri, 19 Apr 2024 at 10:30, Thorsten Kukuk <kukuk@xxxxxxxx> wrote:
>
> > > And now I started looking into how services can survive the
> > > soft-reboot. I know the FOSDEM talk from Luca about this topic, but I
> > > don't like to move the application into another image, as this would
> > > only move the update problem to a different level and not solve it. So
> > > I'm currently playing with it to find out if there isn't a better
> > > option, especially with btrfs.
>
> > It really needs to be a separate filesystem from a separate image, any
> > ties back to the host OS and the service will be hopefully correctly
> > stopped, or worse it will not be detected and it will leak the old
> > filesystem, which means you'll silently leak memory, mounts, etc. I
> > would strongly recommend to avoid fighting against this, and instead
> > spend time solving the root cause.
>
> I agree that you have a problem if you use something like a partition
> A/B setup, where in worst case, you start from A, soft-reboot to B,
> updates A and latest in this moment you will overwrite the code of the
> running application.
> I'm not sure if this is really a problem with btrfs and subvolumes, as
> they stay mounted anyways in our setup.
> I hope I find the time to discuss this next week with our btrfs developers.
>
> > The best solution really is to figure out why there's a executable
> > from the host OS permanently running in the podman container cgroup
> > (what does it do, why it is necessary, why does it need to always run,
> > etc), and try to refactor that away. Make it started on demand for
> > example.
>
> That's conmon, no idea why it needs to continue to run. But this would
> only solve the problem for podman, I have several other use cases in
> mind where containers are not involved.

Having no dependency on the rootfs is a core requirement that we
cannot change, the filesystem used is irrelevant. It's not going to
work, it will break immediately if you are lucky and in subtle and
invisible ways if you are not. Just move the binaries to a different
image as a generic solution, and where possible remove them entirely.
Portable services work great for this purpose.