Re: Root remaining read-only after boot, no obvious reason why

[Zbigniew Jędrzejewski-Szmek <zbyszek@xxxxxxxxx>]

On Tue, May 16, 2023 at 02:37:58PM +0000, Dave Re wrote:
> 
> Hi, Zbyszek,
> 
> The kernel default is to mount root ro, so unless there's some configuration to
> mount or remount it rw, it'll stay ro. Generally this is configured in two
> places 'rw' on the kernel command line and 'rw' in /etc/fstab. Do you have
> one or both of those?
> Yes. In this case, the kernel command line in grub contains the usual "ro" option, but /etc/fstab has an entry for root with "defaults" as the sole option. Default options are theoretically "rw, suid, dev, exec, auto, nouser, and async". To verify, I placed that option list in /etc/fstab for root, and saw the same behavior on reboot, so I eliminated the fstab entry as a potential issue.

Yes, systemd-remount-fs just calls 'mount -o remount /', and that remounts rw if
'ro' has not been specified. So the problem seems to be elsewhere.

> Also related - the device is identified in fstab by UUID, not by label or partition. I verified the UUID of the device still matches the UUID in fstab, etc.

This shouldn't matter. For the remount, we look for the file system by
mountpoint.

> If I change the kernel command line in grub to use "rw", root mounts rw as expected.
> If a remount is done, it'll be done by systemd-remount-fs.service. Do you
> see it in the logs?
> Ah, I misunderstood initrd-switch-root - thanks. On a system that successfully boots with root mounted rw, I don't see anything about systemd-remount-fs.service until it's shutdown, when the service is stopped, but systemd was ratelimited by the kernel on that boot, so a startup message may have been missed. But - I can see that systemd-remount-fs.service is Active/exited (via systemctl status command).
> 
> On the problem system, systemd-remount-fs.service is inactive/dead - and shows the same status if I reboot with "rw" on the kernel command line (which would be expected, given our symptoms, I think?).

OK, so this is the problem. It seems that the service wasn't started at all.

> On both systems, it's linked in /usr/lib/systemd/system/local-fs.target.wants - so, should be run as part of getting to that target, if I understand correctly? What should my next step be to determine why it's not being run?

Yes.

Hmm, so we stopped enabling systemd-remount-fs.service in this way back in 2018
(https://github.com/systemd/systemd/commit/9b69569d2c). At this point,
that's ancient history for us ;)

Zbyszek