Re: Unit shutdown order not always respected

[Lennart Poettering <lennart@xxxxxxxxxxxxxx>]

On Do, 30.06.22 09:46, David Gubler (david.gubler@xxxxxxxx) wrote:

> Hi list,
>
> I have a situation where I need to run a command and wait for its completion
> before unmounting a file system (/enc in my case). My problem is that
> systemd sometimes waits for the completion of the command, and sometimes
> doesn't.
>
> So the setup is:
> * /enc is a mounted encrypted (luks) volume
> * /var/lib/mysql is a bind mount to /enc/mysql
> * MariaDB is using /var/lib/mysql
> * We've set up a "requires" and "after" dependency chain from MariaDB all
> the way to the luks volume. This works 100% reliably during startup, even if
> something takes too long, fails, or if we have to manually fiddle with
> stuff.
> * Ubuntu 20.04 with systemd 245.4
>
>
> The unit of the command that needs to run before unmounting /enc looks like
> this:
>
>
> [Unit]
> Description=server-secrets-prepare-reboot-enc service
> After=network-online.target enc.mount
> Requires=network-online.target
> BindsTo=enc.mount
>
> [Service]
> ExecStart=/bin/true

You can drop this line.

> ExecStop=/usr/local/sbin/server-secrets reboot "/enc"
> Restart=no

This is the implied default.

> StandardOutput=syslog
> StandardError=syslog

These lines are obsolete. Drop them.

> SyslogIdentifier=server-secrets-prepare-reboot-enc
> User=root

This is the impleid default.

> Type=oneshot
> RemainAfterExit=yes
>
> [Install]
> WantedBy=enc.mount
>
>
>
>
> I've added a 10s sleep to the "server-secrets" command in order to eliminate
> "works by chance" situations.
>
> In normal circumstances everything works perfectly. I can reboot the server
> and systemd waits >10s with unmounting /enc until server-secrets is done. I
> can stop mariadb, umount /var/lib/mysql and /enc, and systemd runs
> server-secrets just fine. I can re-mount /enc, unmount it again, everything
> works.
>
> Where things start to fall apart is when I reboot the server ("reboot" or
> "shutdown -r now") after manually stopping MariaDB. In this case systemd
> starts the server-secrets command before unmounting /enc, but does not wait
> for its completion, and immediately unmounts /enc, causing the command to
> fail.
>
> Note that manually stopping MariaDB does not unmount anything, it just...
> stops MariaDB. But somehow it causes systemd to change its ordering behavior
> during shutdown.
>
> Does this ring any bells?

You are starting this service at boot right? Or are you starting it
only during shutdown? Genreally if code shall run during shutdown the
right approach is to make it a service that starts at boot-up, but has
an ExecStart= that is empty, but an ExecStop= set that has a valid command.

Either way, consider having a look at the debug logs, to see what
happens. Usually you probably have some odering cycle between units,
which we'll try to fix for you, but which will of course mean the
ordering is not going to be executed in full.

See:

https://freedesktop.org/wiki/Software/systemd/Debugging/#diagnosingshutdownproblems

Lennart

--
Lennart Poettering, Berlin