Great question!
I am very interested in detecting systemd crashes too since I have experienced them recently and have been asked to come up with a solution to react when a PID1 crash happens.
In fact, in my recent experiences, a journald crash was enough to render the system into an unreliable/degraded state in which some top-level applications worked while others didn't.
So adding to David's 1st question, I need to detect systemd and journald crashes and then trigger a `systemctl reboot --force --force` command
I have also read that Linux Magic System Request Key (SysRq) can help in such scenarios but I don't know how they work.
Any help would be very appreciated.
Thank you.
Some related links:
El sáb, 3 feb 2024 a las 16:14, David Timber (<dxdt@xxxxxxxxxxxx>) escribió:
Systemd crashed on me the other day. I was writing up some Systemd units
and testing them out by daemon-reload every time I wanted to test them
out. Not the best way to go on about, I know. My bad abusing Systemd to
the point of crashing. Perhaps it was just a bit flip that caused this.
systemd[2368]: Assertion 'path_is_absolute(p)' failed at
src/basic/chase.c:628, function chase(). Aborting.
systemd[1]: Assertion 'path_is_absolute(p)' failed at
src/basic/chase.c:628, function chase(). Aborting.
systemd[1]: Caught <ABRT> from our own process.
systemd-coredump[32497]: Due to PID 1 having crashed coredump
collection will now be turned off.
systemd-coredump[32497]: [🡕] Process 32496 (systemd) of user 0
dumped core.
systemd[1]: Caught <ABRT>, dumped core as pid 32496.
systemd[1]: Freezing execution.
...
systemd-journald[871]: Failed to send stream file descriptor to
service manager: Transport endpoint is not connected
I didn't even bother trying producing stack trace. I can get on that if
anyone wants it. My machine started doing some weird things like Firefox
not being able to do Ajax properly whilst being able to go to a new
page, Chromium not being able to create a new tab whilst all the text
editors worked just fine, all the systemctl commands timing out. So
basically, I was using Linux without fork(). Anyway.
Well, I think any software can crash for any reason whatsoever. The
problem with Systemd I realised from this incident is that I had no way
of knowing that Systemd had crashed until I opened up the journal and
kernel logs and saw that Systemd had crashed some time ago. In this
particular incident, Systemd caught the signal and decided to just
freeze. No idea why you'd want that because if it had just crashed, the
kernel would have just panicked and I would have realised something went
wrong.
1: So I decided that I need a some sort of "watchdog" that warns me when
something like this happens. Using dbus to poll the status of the
Systemd process, it could be a GUI app running under a seat, just a
daemon that writes a warning message using `wall` or just send mail
using a primed up MUA process. I wonder if someone already had the same
idea and went on to make one.
2: How do I get Systemd to freeze to test such program? I mean, if I
kill Systemd, the kernel would crash so I have to somehow tell Systemd
to freeze?
