BindsTo and parameterized instance units

[Simon Mullis <simon@xxxxxxxxxxxx>]

Hi All

I have a fairly complex (at least to me) setup of a master target spawning multiple services and groups of instance services that are chained in a specific order. I use systemd to manage all of the sockets that allow data to flow between these different stages.

I use a master Target (foo.target) defined to manage the services state, so I can easily stop and restart everything.

The first service (bar.service) is oneshot script that starts multiple groups of instance services (the number of spawned services depends on CPU cores and queue sizes among other things). I have ExecStart and ExecStop scripts in the unit file.

For example: bar.service - This is the oneshot that spawns "n" baz@.service and "n" qux@.service.  There are a lot of dependencies and so far systemd has done everything I need.

What do I want?

If there is any failure or issue with any of the child processes spawned from any of the instance units then I would like the whole fragile house of cards to be torn down and restarted. i.e. the whole foo.target system state to be restarted, not just the individual instance service (and subsequent process) itself.

What are my observations?

This all works well except for the instance units. When I include the instance units into the "BindsTo" with the target, I get additional processes and services launched that I do not expect.

I have simplified the whole thing to two services, a target and a very simple script. This demonstrates exactly the same thing that I see in the much more complex version.

The "master service". This is a oneshot that spawns the instance units.

foo.service

[Unit]
Description=Foo service
BindsTo=foo.target
[Service]
Type=oneshot
ExecStart=some-path-somewhere/foo-start.sh
[Install]
WantedBy=foo.target

The instance unit that does the actual work. In this case we have a placeholder to show the problem. In my real example I have a long chain of services like this that uses systemd managed sockets to pass data along.

bar@.service

[Unit]
Description=Test service Bar instance %i
BindsTo=foo.target
[Service]
Type=simple
ExecStart=sh -c 'while true; do echo Bar %i is alive; sleep 3; done'
[Install]
WantedBy=foo.target

The target that allows me to stop and restart everything easily:

[Unit]
Description=Test Services
Requires=foo.service
[Install]
WantedBy=multi-user.target

And finally the script called in foo.service:

foo-start.sh

#!/usr/bin/bash
num=4
eval systemctl start bar@{1..${num}}.service

In order to tightly couple the processes and services I use BindsTo. But I am getting inconsistent behavior when trying to apply this to the instance units from the target.

Scenario A:

WITHOUT BindsTo for the instance units in the target:

- Everything stops and starts with the target.

- I get the correct number of processes.

- If I kill one of the PIDs below, systemd only restarts that process - which of course is what most use-cases would require.

# ps -ef | grep [Bb]ar

root       17878       1  0 15:25 ?        00:00:00 sh -c while true; do echo Bar 1 is alive; sleep 3; done
root       17880       1  0 15:25 ?        00:00:00 sh -c while true; do echo Bar 2 is alive; sleep 3; done
root       17882       1  0 15:25 ?        00:00:00 sh -c while true; do echo Bar 3 is alive; sleep 3; done
root       17887       1  0 15:25 ?        00:00:00 sh -c while true; do echo Bar 4 is alive; sleep 3; done

 # systemctl list-units bar@\*.service
  UNIT            LOAD   ACTIVE SUB     DESCRIPTION
  bar@1.service   loaded active running Test service Bar instance 1
  bar@2.service   loaded active running Test service Bar instance 2
  bar@3.service   loaded active running Test service Bar instance 3
  bar@4.service   loaded active running Test service Bar instance 4

Scenario B:

WITH BindsTo in the unit instance file (BindsTo=bar@%i.service or BindsTo=bar@%N.service):

- Everything stops and start with the target.

- i get EXTRA PROCESSES.

- If I kill one of the PIDs below, everything restarts properly (i.e. the whole target) and I get the behavior I am looking for.

root       29250       1  0 16:08 ?        00:00:00 sh -c while true; do echo Bar foo is alive; sleep 3; done  #<<<< What's this guy doing here?
root       29256       1  0 16:08 ?        00:00:00 sh -c while true; do echo Bar 1 is alive; sleep 3; done
root       29258       1  0 16:08 ?        00:00:00 sh -c while true; do echo Bar 2 is alive; sleep 3; done
root       29260       1  0 16:08 ?        00:00:00 sh -c while true; do echo Bar 3 is alive; sleep 3; done
root       29262       1  0 16:08 ?        00:00:00 sh -c while true; do echo Bar 4 is alive; sleep 3; done

So, however systemd is expanding the variables %i or %N, it's including an additional service.

 # systemctl list-units bar@\*.service
  UNIT            LOAD   ACTIVE SUB     DESCRIPTION
  bar@1.service   loaded active running Test service Bar instance 1
  bar@2.service   loaded active running Test service Bar instance 2
  bar@3.service   loaded active running Test service Bar instance 3
  bar@4.service   loaded active running Test service Bar instance 4
  bar@foo.service loaded active running Test service Bar instance foo   #<<<< Here he is again! ????

Does anyone have any suggestions?  Is there a more elegant way to connect the processes to the whole target for restart purposes?

Maybe this is a bug in my version of systemd but more likely I'm doing something wrong.

Version info:

# systemctl --version
systemd 247 (247.3-7+deb11u1)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified

Thank you for reading this far and thank you also in advance for any suggestions.

Cheers!