Ordering of oneshot services and path units?

[John Ioannidis <systemd-devel@xxxxxxx>]

tl;dr: a .path unit does not appear to be waiting for the After= unit to run first.

I am still trying to understand why some services occasionally do not start at boot time. It is a very intermittent behavior, but I caught another instance. Everything is running in Google Compute Engine or Amazon EC2. 

I have a one-shot service, mktags.service, a long-running job of type notify, ccistated.service, and a path unit with its corresponding service, workdir.path and workdir.service. Here are the relevant parts of the unit files:

mktags.service

[Unit]

Description=Populate /run/metadata/tags

After=network.target

[Service]

ExecStart=/usr/local/sbin/mktags.py

Type=oneshot

[Install]

WantedBy=multi-user.target

ccistated.service

[Unit]

Description=State Machine Manager

After=mktags.service

ConditionPathExists=/run/metadata/tags/resource_class

[Service]

ExecStart=/usr/local/sbin/ccistated.py

Type=notify

NotifyAccess=all

[Install]

WantedBy = multi-user.target

workdir.path

[Unit]

Description=Trigger workdir.service when a job starts, creating a directory in /opt/circleci/workdir

After=ccistated.service

ConditionPathExists=/run/metadata/tags/resource_class

[Path]

PathChanged=/opt/circleci/workdir

[Install]

WantedBy=multi-user.target

What mktags does is extract the metadata tags from the metadata service, and populate /run/metadata/tags/. 

$ systemctl status mktags.service

* mktags.service - Populate /run/metadata/tags
    Loaded: loaded (/etc/systemd/system/mktags.service; enabled; vendor preset: enabled)
    Active: inactive (dead) since Sat 2021-03-27 05:34:00 UTC; 10min ago
   Process: 454 ExecStart=/usr/local/sbin/mktags.py (code=exited, status=0/SUCCESS)
  Main PID: 454 (code=exited, status=0/SUCCESS)

Sure enough, it ran:

$ cat /run/metadata/tags/resource_class
waw

As did the next service:

$ systemctl status ccistated.service
* ccistated.service - State Machine Manager
    Loaded: loaded (/etc/systemd/system/ccistated.service; enabled; vendor preset: enabled)
    Active: active (running) since Sat 2021-03-27 05:36:00 UTC; 8min ago
  Main PID: 1420 (python3)
     Tasks: 1 (limit: 9544)
    Memory: 8.5M
    CGroup: /system.slice/ccistated.service
            `-1420 python3 /usr/local/sbin/ccistated.py

It's OK that it took a couple of minutes for ccistated to start up, it does a few things before sending out the Notify.

But look at the old workdir.path:

$ systemctl status workdir.path
* workdir.path - Trigger workdir.service when a job starts, creating a directory in /opt/circleci/workdir
    Loaded: loaded (/etc/systemd/system/workdir.path; enabled; vendor preset: enabled)
    Active: inactive (dead)
  Triggers: * workdir.service
 Condition: start condition failed at Sat 2021-03-27 05:33:59 UTC; 22min ago
            `- ConditionPathExists=/run/metadata/tags/resource_class was not met

Huh?!?!?! It's supposed to run after ccistated, and of course after mktags (highlighted in cyan above). It appears to be running anyway, and of course mktags has not gotten a chance to finish, and the ConditionExists file has not been created yet.

Do .path units not obey the same startup rules?

_______________________________________________
systemd-devel mailing list
systemd-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/systemd-devel