How to debug systemd services failing to start with 11/SEGV?

[Alexander Dahl <ada@xxxxxxxxxxx>]

Hello everyone,

I'm currently trying to build a firmware for an embedded device and
running into trouble because systemd seems to crash.  The BSP is
based on pengutronix DistroKit (master) built with ptxdist and the
target is the Microchip SAM9X60-Curiosity board, which is arm v5te
architecture (that board is not part of DistroKit, support for that is
in an upper layer of mine not public yet (?)).

Everything is quite recent, building systemd version 255.2 currently.
On startup I get messages like this (this is the first one, later on
there are lot more, all with the same status):

   [   11.175650] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=11/SEGV
   [   11.239679] systemd[1]: systemd-journald.service: Failed with result 'signal'.
   [   11.292640] systemd[1]: Failed to start systemd-journald.service.
   [FAILED] Failed to start systemd-journald.service.
   See 'systemctl status systemd-journald.service' for details.

The system drops me on a shell later, where I can run the above
mentioned command, which gives:

    ~ # systemctl status systemd-journald.service
    x systemd-journald.service - Journal Service
         Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static)
         Active: failed (Result: signal) since Tue 2024-04-09 11:44:52 UTC; 11min a>
    TriggeredBy: x systemd-journald-dev-log.socket
                 * systemd-journald-audit.socket
                 x systemd-journald.socket
           Docs: man:systemd-journald.service(8)
                 man:journald.conf(5)
        Process: 197 ExecStart=/usr/lib/systemd/systemd-journald (code=killed, sign>
       Main PID: 197 (code=killed, signal=SEGV)
       FD Store: 0 (limit: 4224)
            CPU: 330ms

This does not help me much.  Other services crashing: systemd-udevd
and systemd-timesyncd, also with status 11/SEGV which is segmentation
fault, right?

I had this board running with an older version of systemd, but I can
not remember which was the last good version.

Could anyone give me a hint please how to debug this?

(The same version runs fine btw. when building that BSP for an arm v7a
target!)

Greets
Alex