Hi,
I am currently troubleshooting a problem where a raspberry pi zero w will not reboot after running the `reboot` command (it does reboot with `reboot -f` however) - it just "hangs". After enabling the serial console, my best guess was that a job is stuck in systemd, so I tried enabling the early boot shell. However, after I run `reboot` in a normal shell and change to the early boot shell, running the command `systemctl` (for example, `systemctl list-jobs`) just hangs forever. I can ctrl+c out of it and run other commands (`ps`, `ls`, etc) but I can't seem to query systemd through systemctl. Is there some way I can view the current state of systemd without this command?
Some further details:
Linux kernel 5.15.89+ running on raspberry pi zero w (via Raspberry Pi OS)
systemd version: systemd 247 (247.3-7+rpi1+deb11u1)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified
systemd arguments on kernel commandline: systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M enforcing=0 ignore_loglevel systemd.debug-shell=1
Here is the tail end of the systemd log which is output to the serial console after running `reboot`:
```
[ 126.857450] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 126.942054] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.213116] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.272100] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.318344] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.522280] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 128.217070] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/bluetooth_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=799 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.274042] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/bluetooth_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=800 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.326185] systemd[1]: Child 522 (agetty) died (code=killed, status=15/TERM)
[ 128.347548] systemd[1]: getty@tty1.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 128.371639] systemd[1]: getty@tty1.service: Child 522 belongs to getty@tty1.service.
[ 128.390389] systemd[1]: getty@tty1.service: Main process exited, code=killed, status=15/TERM
[ 128.406372] systemd[1]: getty@tty1.service: Succeeded.
[ 128.423193] systemd[1]: getty@tty1.service: Service restart not allowed.
[ 128.438162] systemd[1]: getty@tty1.service: Changed stop-sigterm -> dead
[ 128.457233] systemd[1]: getty@tty1.service: Job 662 getty@tty1.service/stop finished, result=done
[ 128.466383] systemd[1]: Stopped Getty on tty1.
[ 128.499079] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=801 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.539439] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=802 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.575152] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=803 reply_cookie=0 signature=uoss error-name=n/a error-message=n/a
[ 128.610134] systemd[1]: getty@tty1.service: Consumed 40ms CPU time.
[ 128.635696] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=804 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.672136] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=805 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.713716] systemd[1]: Child 525 ((agetty)) died (code=killed, status=15/TERM)
[ 128.737541] systemd[1]: getty@ttyGS0.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 128.758022] systemd[1]: getty@ttyGS0.service: Child 525 belongs to getty@ttyGS0.service.
[ 128.766575] systemd[1]: getty@ttyGS0.service: Main process exited, code=killed, status=15/TERM
[ 128.779496] systemd[1]: getty@ttyGS0.service: Succeeded.
[ 128.785390] systemd[1]: getty@ttyGS0.service: Service restart not allowed.
[ 128.794270] systemd[1]: getty@ttyGS0.service: Changed stop-sigterm -> dead
[ 128.804306] systemd[1]: getty@ttyGS0.service: Job 700 getty@ttyGS0.service/stop finished, result=done
[ 128.814886] systemd[1]: Stopped Getty on ttyGS0.
[ 128.823697] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40ttyGS0_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=806 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.853005] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40ttyGS0_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=807 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.881151] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=808 reply_cookie=0 signature=uoss error-name=n/a error-message=n/a
[ 128.904528] systemd[1]: getty@ttyGS0.service: Consumed 7ms CPU time.
[ 201.747274] systemd-journald[100]: Sent WATCHDOG=1 notification.
[ 301.957264] systemd-journald[100]: Sent WATCHDOG=1 notification.
[ 126.942054] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.213116] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.272100] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.318344] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 127.522280] systemd-journald[100]: Successfully sent stream file descriptor to service manager.
[ 128.217070] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/bluetooth_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=799 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.274042] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/bluetooth_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=800 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.326185] systemd[1]: Child 522 (agetty) died (code=killed, status=15/TERM)
[ 128.347548] systemd[1]: getty@tty1.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 128.371639] systemd[1]: getty@tty1.service: Child 522 belongs to getty@tty1.service.
[ 128.390389] systemd[1]: getty@tty1.service: Main process exited, code=killed, status=15/TERM
[ 128.406372] systemd[1]: getty@tty1.service: Succeeded.
[ 128.423193] systemd[1]: getty@tty1.service: Service restart not allowed.
[ 128.438162] systemd[1]: getty@tty1.service: Changed stop-sigterm -> dead
[ 128.457233] systemd[1]: getty@tty1.service: Job 662 getty@tty1.service/stop finished, result=done
[ 128.466383] systemd[1]: Stopped Getty on tty1.
[ 128.499079] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=801 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.539439] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=802 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.575152] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=803 reply_cookie=0 signature=uoss error-name=n/a error-message=n/a
[ 128.610134] systemd[1]: getty@tty1.service: Consumed 40ms CPU time.
[ 128.635696] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=804 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.672136] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40tty1_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=805 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.713716] systemd[1]: Child 525 ((agetty)) died (code=killed, status=15/TERM)
[ 128.737541] systemd[1]: getty@ttyGS0.service: Failed to read oom_kill field of memory.events cgroup attribute: No such file or directory
[ 128.758022] systemd[1]: getty@ttyGS0.service: Child 525 belongs to getty@ttyGS0.service.
[ 128.766575] systemd[1]: getty@ttyGS0.service: Main process exited, code=killed, status=15/TERM
[ 128.779496] systemd[1]: getty@ttyGS0.service: Succeeded.
[ 128.785390] systemd[1]: getty@ttyGS0.service: Service restart not allowed.
[ 128.794270] systemd[1]: getty@ttyGS0.service: Changed stop-sigterm -> dead
[ 128.804306] systemd[1]: getty@ttyGS0.service: Job 700 getty@ttyGS0.service/stop finished, result=done
[ 128.814886] systemd[1]: Stopped Getty on ttyGS0.
[ 128.823697] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40ttyGS0_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=806 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.853005] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/getty_40ttyGS0_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=807 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
[ 128.881151] systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=808 reply_cookie=0 signature=uoss error-name=n/a error-message=n/a
[ 128.904528] systemd[1]: getty@ttyGS0.service: Consumed 7ms CPU time.
[ 201.747274] systemd-journald[100]: Sent WATCHDOG=1 notification.
[ 301.957264] systemd-journald[100]: Sent WATCHDOG=1 notification.
```
The "Sent WATCHDOG=1 notification." is just printed over and over as long as I leave the system on (I left it running for 12hrs and it did not reboot, it just printed this message). At this point in the shutdown, the early shell is still accessible, but the `systemctl` command just hangs.
I have also posted the full log here as an attachment: https://github.com/raspberrypi/firmware/issues/1559#issuecomment-1403052508
Any pointers you could provide me in further debugging this problem would be greatly appreciated. I am not exactly sure where to go from here seeing as the debug shell seems to be not working for me.
Thanks,
Jeremy
