Re: Systemd hang when restarting a service during shutdown

[Michael Chapman <mike@xxxxxxxxxxxxxxxxx>]

On Mon, 8 Nov 2021, Sean Nyekjaer wrote:
> Hi,
> 
> Regarding,
> https://github.com/systemd/systemd/issues/21203
> 
> I think the point of the issue missed when the issue got closed.
> 
> We have a service that is changing configs for systemd-networkd and
> issuing a `systemctl restart systemd-networkd`.
> An other service is checking uptime and issues a `systemctl reboot`,
> when our max uptime have been exeeced.
> If restart of systemd-networkd happens while a reboot is in progress,
> the system will hang "forever" (and continue to pet the watchdog).
> This is not a thing that eventually will timeout and reboot the
> system...
> 
> /Sean

This looks very similar to a problem I saw mentioned on Reddit a few days 
ago:

   https://www.reddit.com/r/Fedora/comments/qgj1bh/systemctl_emergency_question_f34_f35_beta/

There the problem was that `systemctl emergency` hung. It looked like it 
was because the transaction to switch to emergency mode got cancelled, as 
something wanted to talk to logind after it had shut down. Socket 
activation started it back up again, and that cancelled the remaining jobs 
in the transaction. systemd ended up idle with most services shut down and 
with nothing useful running.

In your case it's not logind that's being restarted... it's resolved or 
networkd. But the overall effect appears to be the same. You end up with a 
system where most things are stopped, and where it's not making any 
progress towards the reboot you asked for.