Looking for guidance about starting a systemd service inside the initrd and having it persist after rootfs is mounted

[Brian Masney <bmasney@xxxxxxxxxx>]

I am working on a project that has very strict boot time requirements
in order to have a custom service started within a set time period.
Waiting for the kernel to initialize the storage controller and mount
the root filesystem takes too much of the allocated time budget.
There's various boot speed optimizations that we are working on and
it's going to take a combination of multiple approaches.

One thing we are investigating is to start the time-critical service
inside the initrd with the following systemd unit:

    [Unit]
    Description=early service example
    DefaultDependencies=no
    Before=sysinit.target
    IgnoreOnIsolate=yes
    
    [Service]
    ExecStart=/usr/bin/some-service
    Restart=always
    
    [Install]
    WantedBy=sysinit.target
    WantedBy=initrd-switch-root.target
    WantedBy=multi-user.target
    WantedBy=graphical.target

I also have a dracut module that copies the service binary and
associated systemd unit into the intrd. We also need to set
argv[0][0] = '@' as described at
https://systemd.io/ROOT_STORAGE_DAEMONS/ to prevent systemd from
sending a SIGTERM to the process when it pivots to the root
filesystem. However, the page says "This all applies to storage
technology only, not to daemons with any other (non-storage related)
purposes.", which the use cases we are looking at are not storage
related.

Does anyone have any suggestions? It seems like the root storage
daemon is what we should use right now.

Ideally it seems that we should have a relatively short-lived process
that's started from the initrd, and then start up another copy when
the root filesystem is started, and pass on the state of the service
to the main copy so that the initrd version can terminate. That's
going to be difficult refactor.

Here's a few of the drawbacks that I've come up with to starting
something in the initrd and having it persist while the main system
is running:

- Need to signal to the process after the root filesystem is mounted
  that it needs to also pivot to /sysroot.
- The long-running process will prevent the initrd filesystem from being
  unmounted.
- Limited size available in the initrd.
- Can't use User/Group in the unit file with xxx-sysusers.conf since
  the users won't exist at this point.
- Page cache for shared libraries won't be shared with processes on the
  root filesystem.
- If the process terminates for some reason after systemd has switched
  to the root filesystem, we have 'Restart=always', and systemd will
  potentially be starting a different binary. Ideally the binary on the
  root filesystem should match what's in the initrd, but that won't
  always be the case. For example, the initrd for older kernels won't be
  rebuilt.
- Any other drawbacks?

Brian