On Fri, Apr 20, 2018 at 6:15 AM, Florian Weimer <fweimer@xxxxxxxxxx> wrote: > I'm trying to pin down what exposed this bug: > > https://bugzilla.redhat.com/show_bug.cgi?id=1569970 > > The immediate trigger seems to be that all shutdowns on my system leave the > XFS root file system in an unclean state, so that GRUB cannot read recently > written files under /boot (assuming that /boot is on the same file system). > > Since GRUB does not deal with the XFS journal, the effect is that the system > is unbootable until you mount / and trigger journal recovery, after which > everything is fine again (because the files in /boot are only written to > rarely). > > Any ideas what could cause this, and how to debug this further? It's > clearly GRUB's fault that it cannot read the XFS journal, but the > consistently unclean shutdowns probably have been caused by a kernel or > systemd change, or maybe something else entirely. It's sabotage. The characteristics leading up to this sabotage: a. User and installer, innocently ignorant of the sabotage, put the bootloader configuration on rootfs (i.e. /boot is not a separate fs). b. grubby, grub-mkconfig do not guarantee the modified bootloader configuration is fully committed to disk (flushed from the log) c. GRUB doesn't support log replay, which is why it is obligated under b. to flush it's shit correctly d. systemd uses the honor system to allow services to exempt themselves from being killed, plymouth exempts itself in this commit https://cgit.freedesktop.org/plymouth/commit/?id=9e5a276f322cfce46b5b2ed2125cb9ec67df7e9f That leads to systemd neither cleanly umounting rootfs nor remounting read-only, which means the log is not flushed when it forcibly reboots anyway. Result: the grub.cfg can't be read by the bootloader and now the user is betrayed. If any of those four things are false, the problem doesn't happen. The unscientific sampling suggests XFS is more prone to the problem than ext4, but switching file systems or a file system specific ioctl is in my view inadequate. Btrs seems immune, but in theory it's possible you'd see the bootloader find a stale bootloader configuration which boots the previous kernel, log replays, and at next reboot the bootloader sees new bootloader config and kernel. I'm honestly mystified why the plymouth commit hasn't been reverted in the interim. But I'm also mystified why the bootloader folks don't give a shit to commit their configuration files to disk when they know they can't do journal replay and have known that for 20 years. But then I'm also mystified why systemd developers won't fallback to freeze/thaw if rootfs remount-ro fails three times, instead of just giving up and forcing a reboot, they did discuss doing this a year ago and then poof, no action. -- Chris Murphy _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
