Fwd: What could be causing a oneshot unit not to be run when a machine reboots?

[John Ioannidis <systemd-devel@xxxxxxx>]

I have the following service file:

[Unit]

Description=D

After=network.target

[Service]

ExecStart=/usr/local/sbin/mktags.py

Type=oneshot

User=root

[Install]

WantedBy=multi-user.target

Occasionally, when the machine reboots, it does not run. Here is the evidence:

# uptime
05:50:11 up 20 min,  1 user,  load average: 0.14, 0.03, 0.01

The machine has been up for 20 minutes, so it rebooted at 05:30

# systemctl status mktags.service

* mktags.service - D
    Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor preset: enabled)

    Active: inactive (dead)

Weird... no indication of what the last process was... let's check journalctl:

# journalctl -u mktags.service | tail -4

-- Reboot --

Mar 14 05:25:22 sll9 systemd[1]: Starting D...

Mar 14 05:25:22 sll9 systemd[1]: mktags.service: Succeeded.

Mar 14 05:25:22 sll9 systemd[1]: Finished D.

The last time the service ran was five minutes before I rebooted. Other times the service runs fine. On the same machine, a bit later:

# uptime
05:57:31 up 2 min,  1 user,  load average: 0.31, 0.33, 0.14

# systemctl status mktags.service
* mktags.service - D
    Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor preset: enabled)
    Active: inactive (dead) since Sun 2021-03-14 05:55:14 UTC; 2min 24s ago
   Process: 468 ExecStart=/usr/local/sbin/mktags.py (code=exited, status=0/SUCCESS)
  Main PID: 468 (code=exited, status=0/SUCCESS)
(more output indicated that the process indeed ran).

How do I even debug something like this? I have added code in mktags.py to print stuff to stderr, so that journalctl picks it up. When the service runs, the expected output is there. Where it does not run, it is not there of course.

I cannot detect any other errors elsewhere; if the service fails to start, and I reboot, it invariably runs then. Obviously this is not a solution; these are unattended machines running on GCE!

_______________________________________________
systemd-devel mailing list
systemd-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/systemd-devel