Re: Reg: systemd boot up hung after sysroot.mount in initrd phase

[RAJESH DASARI <raajeshdasari@xxxxxxxxx>]

On Mon, May 15, 2023 at 10:34 AM Lennart Poettering
<lennart@xxxxxxxxxxxxxx> wrote:
>
> On So, 07.05.23 20:28, RAJESH DASARI (raajeshdasari@xxxxxxxxx) wrote:
>
> > Hi ,
> >
> > We are issuing reboot with this command  systemctl  --no-block reboot
> > and  boot up is stuck at initrd phase , we are using dracut and
> > systemd in initramfs phase. Boot up is stuck after  sysroot.mount is
> > executed(Issue is always reproducible), I have enabled debug logs by
> > passing systemd.log_level=debug systemd.log_target=console to kernel
> > command line , issue is disappeared and I also tried to enable the
> > dracut debug logs rd.debug=1, issue is disappeared. Then I have added
> > sleep 1 in the systemd-fsck-root.service , then the issue is not seen.
> > So it looks like a timing issue.
> >
> >
> > Starting [0;1;39msystemd-fsck-root.service[0m - File System Check on
> > /dev/disk/by-label/rootfs...
> > Finished [0;1;39msystemd-fsck-root.service[0m - File System Check on
> > /dev/disk/by-label/rootfs.
> > Mounting [0;1;39msysroot.mount[0m - /sysroot...
> >  EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
> >  Mounted [0;1;39msysroot.mount[0m - /sysroot. -> boot up stuck after this.
> >
> >
> > We are using systemd version-251.13 and with systemd  version-249.14
> > we didn't notice this. Is there any new issue with this version or Is
> > there any known issue if not Could you please let me know how to debug
> > this further.
>
> This is a very vague report. It doesn't specify which distro is even
> used. With the short logs excerpt one cannot figure anything out.
>
> This looks a lot like an integration issue, i.e. something you shoul
> first ask your distro about.
>
> Other than that, there's this:
>
> https://freedesktop.org/wiki/Software/systemd/Debugging/#diagnosingbootproblems
>
> Lennart
>
yes I agree that it is a vague report, Unfortunately I can't provide
much logs also if the debug logs are enabled the issue is not seen. I am mostly
looking for some inputs on debugging this further.
since boot up is getting hung after sysroot.mount is mounted(mounting
is successful and no file system related errors and no timeouts), and
adding a delay of 1 second is fixing this issue, so it  is hard for me
to find where it is getting hung as i am not fully aware of the
systemd execution flow at initramfs phase.
Few symptoms of this issue are :
1) Issue is seen only when the VM is rebooted with systemctl
--no-block reboot. (VM launched from openstack and it is based on
fedora 37)
2) When we do the soft reboot multiple times issue is still present
and VM is hung at bootup, but after a hard reboot node is able to boot
up normally.
3) Issue gets disappeared with one second of sleep in
systemd-fsck-root.service  or  also if the debug logs are enabled from
either systemd or dracut.
To me it looks like a race condition, but not sure on how to debug further.
> --
> Lennart Poettering, Berlin