hi,
I have a problem.... I'm not entirely sure where to fix it.
I have a system with 2 drives, each partitioned into 3 partitions and those
partitions combined in md/raid RAID1 arrays.
The arrays are used for /boot, swap and an LVM PV - which has the root
filesystem in a VG. Normally this all works wonderfully.
If I shutdown, remove one drive, and boot - it doesn't.
When mdadm sees a newly-degraded array like this (i.e. the one drive it
finds doesn't have recorded that the other device is missing), it won't
assemble the array until it is explicitly told that all devices have been
found.
dracut has code to do exactly this: /sbin/mdraid_start is placed on the
'timeout' queue to be run after a suitable timeout. And this script does
the right thing. The md arrays are all assembled, the LVM PV is found so
the VG and LV is assembled, and the root filesystem is mounted..
HOWEVER, before all that happens, systemd stops waiting and gives up -
moments too soon.
There are (at least) two other scripts on the 'timeout' queue which run
immediately after mdraid_start.
One is lvmscan, which probably does the right thing, isn't exactly necessary
in my context I think, and certainly isn't a problem.
But there is also the 'resume.sh' script:
{
printf -- "%s" 'warn "Cancelling resume operation. Device not found.";'
printf -- ' cancel_wait_for_dev /dev/resume; rm -f -- "$job" "%s/initqueue/settled/resume.sh";\n
} >> $hookdir/initqueue/timeout/resume.sh
This calls cancel_wait_for_dev, which supposedly cancels the wait for the
swap array. Unfortunately it also cancels the wait for the root filesystem.
I'm not completely sure why, but it certainly relates to the
systemctl daemon-reload
command that cancel_wait_for_dev schedules. If I comment that out, it all
works. (also if I boot with noresume, it all works).
But I don't think that is all of the problem.
If I had swap on an LVM volume, then I suspect it wouldn't quite be found by
the time that the resume.sh script gets run, and so the resume attempt would
incorrectly abort.
My feeling is that if any script in the 'timeout' queue makes any progress,
then the remaining scripts should be delayed for another timeout.
There is code that does something a little bit like this:
if [ $main_loop -gt $((2*$RDRETRY/3)) ]; then
for job in $hookdir/initqueue/timeout/*.sh; do
[ -e "$job" ] || break
job=$job . $job
udevadm settle --timeout=0 >/dev/null 2>&1 || main_loop=0
[ -f $hookdir/initqueue/work ] && main_loop=0
done
fi
so if 'udevadm settle --timeout=0' fails, or if initqueue/work has been
created, the main_loop is set to 0, which seems to imply "try again".
However the subsequent jobs in the queue are not aborted.
Also, in my case, neither of these conditions trigger. mdraid_start doesn't
create the 'initqueue/work' file, and 'udevadm settle' has never (as far as I
can tell) actually honoured "--timeout=0" the way it is documented.
It waits indefinitely for the queue to be empty, then succeeds.
So my proposed solution (which is really just a suggestion and I suspect
something else will be better), is to:
1/ modify the above loop to break out if main_loop is ever reset to zero, and
2/ modify mdraid_start to create initqueue/work if it finds anything to do.
The following patch makes that explicit. It seems to fix my problem.
Is this a good way to fix it? Is there something better?
BTW, I also have a problem in a similar config where the md/raid RAID1 is
encrypted. Systemd gives up waiting for the encrypted device after 90
seconds:
[ 92.250437] linux systemd[1]: Dependency failed for Cryptography Setup for cr_md1.
but mdraid_start doesn't get run until 120 seconds have elapsed.
I haven't looked into setup of encrypted devices yet, but if anyone has
suggestions, I'm very interested :-)
Thanks,
NeilBrown
diff --git a/modules.d/90mdraid/mdraid_start.sh b/modules.d/90mdraid/mdraid_start.sh
index 761e64f312d3..400ab5dc46c7 100755
--- a/modules.d/90mdraid/mdraid_start.sh
+++ b/modules.d/90mdraid/mdraid_start.sh
@@ -27,6 +27,7 @@ _md_force_run() {
_path_d="${_path_s%/*}/degraded"
[ ! -r "$_path_d" ] && continue
+ > $hookdir/initqueue/work
done
}
diff --git a/modules.d/98systemd/dracut-initqueue.sh b/modules.d/98systemd/dracut-initqueue.sh
index 88cd1e056ed7..af9cec2c5b8c 100755
--- a/modules.d/98systemd/dracut-initqueue.sh
+++ b/modules.d/98systemd/dracut-initqueue.sh
@@ -60,6 +60,7 @@ while :; do
job=$job . $job
udevadm settle --timeout=0 >/dev/null 2>&1 || main_loop=0
[ -f $hookdir/initqueue/work ] && main_loop=0
+ [ $main_loop -eq 0 ] && break
done
fi
Attachment:
pgpy0RpVvz3QK.pgp
Description: OpenPGP digital signature
