Starting transient services securely from other service without root

[Vašek Šraier <vaclav.sraier@xxxxxx>]

Hi,

TL;DR: I want to start transient system services from another system
service via DBus. All services should have as little privileges as
possible, definitely not root. How can I do that securely?

-----

**Description of a general setup**

Imagine a scenario with a multiprocess service with a master process
and several worker processes. The worker processes in this case are
rather independent - they have their own human-readable log, they
should be restarted if they fail. We do not start/stop them often.
There is almost no IPC between master and workers.

The master proces pretty much just reads configuration and decides on
which and how many workers need to be run. It also provides a way to
reload the configuration.

With these requirements, it seems rather straight-forward to use
systemd to manage the worker processes. The master proces communicates
with systemd via DBus and starts/stops/restarts the workers as needed.
Because the workers are slightly different (e.g. command line args) and
because it could be confusing to admins, we decided to use transient
services so that the workers can't be started without the master
process. The master process is has standard service unit file.

So, if everything is running as expected, the system could look like
this:

      ------------------------------------
      |          systemd                 |
      ------------------------------------
        |                 |          |
        |                 |          |
      master           worker1     worker2

    service           transient   transient
with unit file         service     service



**Reasoning and more details behind the design decisions**

This section can be safely skipped. It's here just to provide more
context for the description above.

We ended up with the above described architecture while working on Knot
Resolver (DNS resolver) at CZ.NIC. The resolver in the current release
is a single-threaded process called kresd. When run on multi-core
system, the administrator is supposed to start multiple instances of
kresd@.service to fully utilize all cores and increase reliability.

Administrators using Knot Resolver have been writing custom shell
scripts to perform rolling restarts after updating configuration and to
perform other routine administrative tasks. We would like to avoid
that. kresd already contains and depends on systemd integration, so the
decision to write the "master process" using systemd's DBus API for
managing the services made a lot of sense. After some experiments, we
settled on using transient services as there was less space for human
error.



**The problem / question**

With the described system, I would like to run master and worker
processes as a non-root user X. Which leads to the core of the problem
- process with non-root priviledges needs to be able to start some
systemd services which themselves do not strictly require root.

The solution must also be something worthy of deploying in production,
it should be possible to make distro packages not requiring any manual
modifications after installing.

My question is then, how can I securely start the worker processes? Any
solutions and opinions are welcome. :)



**Possible leads**

I have researched this problem and I came up with 3 possible ways how
to solve it. I don't see a clear winner though and I am not sure all of
them are even feasible.


- Polkit
  We could create a rule allowing user X to start services also running
  under user X. Essentially improving up on this:
https://wiki.archlinux.org/title/Polkit#Allow_management_of_individual_systemd_units_by_regular_users
  I am not sure this is even possible to do. And I personally don't 
  like the solution as it seems really complex. It would be easy to 
  miss something important and leave some vulnerability in the system.

- User sessions
  The master process and worker processes can also run in a user 
  session. This directly solves problems with privileges. However, I am
  not sure if running a user session with the semantics of a system
  service is possible or a good idea. I also don't know if there is any
  documentation related to user sessions without physical users.

- Use other service managers, not systemd
  Using systemd facilities for logging, restarting and more is
helpful, 
  but what we are trying to do could be different from systemd 
  goals and we can't therefore use systemd like we want to securely.
  With a different service manager (like supervisord) running within 
  the scope of one service unit would also solve the privilege issues.
  This seems like a duplication of effort and it also trashes our 
  existing libsystemd integration within the worker process (kresd).


I would be happy for any input you can provide.
Thank you,
Vašek Šraier
(Knot Resolver @ CZ.NIC)