hi, I have a problem.... I'm not entirely sure where to fix it. I have a system with 2 drives, each partitioned into 3 partitions and those partitions combined in md/raid RAID1 arrays. The arrays are used for /boot, swap and an LVM PV - which has the root filesystem in a VG. Normally this all works wonderfully. If I shutdown, remove one drive, and boot - it doesn't. When mdadm sees a newly-degraded array like this (i.e. the one drive it finds doesn't have recorded that the other device is missing), it won't assemble the array until it is explicitly told that all devices have been found. dracut has code to do exactly this: /sbin/mdraid_start is placed on the 'timeout' queue to be run after a suitable timeout. And this script does the right thing. The md arrays are all assembled, the LVM PV is found so the VG and LV is assembled, and the root filesystem is mounted.. HOWEVER, before all that happens, systemd stops waiting and gives up - moments too soon. There are (at least) two other scripts on the 'timeout' queue which run immediately after mdraid_start. One is lvmscan, which probably does the right thing, isn't exactly necessary in my context I think, and certainly isn't a problem. But there is also the 'resume.sh' script: { printf -- "%s" 'warn "Cancelling resume operation. Device not found.";' printf -- ' cancel_wait_for_dev /dev/resume; rm -f -- "$job" "%s/initqueue/settled/resume.sh";\n } >> $hookdir/initqueue/timeout/resume.sh This calls cancel_wait_for_dev, which supposedly cancels the wait for the swap array. Unfortunately it also cancels the wait for the root filesystem. I'm not completely sure why, but it certainly relates to the systemctl daemon-reload command that cancel_wait_for_dev schedules. If I comment that out, it all works. (also if I boot with noresume, it all works). But I don't think that is all of the problem. If I had swap on an LVM volume, then I suspect it wouldn't quite be found by the time that the resume.sh script gets run, and so the resume attempt would incorrectly abort. My feeling is that if any script in the 'timeout' queue makes any progress, then the remaining scripts should be delayed for another timeout. There is code that does something a little bit like this: if [ $main_loop -gt $((2*$RDRETRY/3)) ]; then for job in $hookdir/initqueue/timeout/*.sh; do [ -e "$job" ] || break job=$job . $job udevadm settle --timeout=0 >/dev/null 2>&1 || main_loop=0 [ -f $hookdir/initqueue/work ] && main_loop=0 done fi so if 'udevadm settle --timeout=0' fails, or if initqueue/work has been created, the main_loop is set to 0, which seems to imply "try again". However the subsequent jobs in the queue are not aborted. Also, in my case, neither of these conditions trigger. mdraid_start doesn't create the 'initqueue/work' file, and 'udevadm settle' has never (as far as I can tell) actually honoured "--timeout=0" the way it is documented. It waits indefinitely for the queue to be empty, then succeeds. So my proposed solution (which is really just a suggestion and I suspect something else will be better), is to: 1/ modify the above loop to break out if main_loop is ever reset to zero, and 2/ modify mdraid_start to create initqueue/work if it finds anything to do. The following patch makes that explicit. It seems to fix my problem. Is this a good way to fix it? Is there something better? BTW, I also have a problem in a similar config where the md/raid RAID1 is encrypted. Systemd gives up waiting for the encrypted device after 90 seconds: [ 92.250437] linux systemd[1]: Dependency failed for Cryptography Setup for cr_md1. but mdraid_start doesn't get run until 120 seconds have elapsed. I haven't looked into setup of encrypted devices yet, but if anyone has suggestions, I'm very interested :-) Thanks, NeilBrown diff --git a/modules.d/90mdraid/mdraid_start.sh b/modules.d/90mdraid/mdraid_start.sh index 761e64f312d3..400ab5dc46c7 100755 --- a/modules.d/90mdraid/mdraid_start.sh +++ b/modules.d/90mdraid/mdraid_start.sh @@ -27,6 +27,7 @@ _md_force_run() { _path_d="${_path_s%/*}/degraded" [ ! -r "$_path_d" ] && continue + > $hookdir/initqueue/work done } diff --git a/modules.d/98systemd/dracut-initqueue.sh b/modules.d/98systemd/dracut-initqueue.sh index 88cd1e056ed7..af9cec2c5b8c 100755 --- a/modules.d/98systemd/dracut-initqueue.sh +++ b/modules.d/98systemd/dracut-initqueue.sh @@ -60,6 +60,7 @@ while :; do job=$job . $job udevadm settle --timeout=0 >/dev/null 2>&1 || main_loop=0 [ -f $hookdir/initqueue/work ] && main_loop=0 + [ $main_loop -eq 0 ] && break done fi
Attachment:
pgpy0RpVvz3QK.pgp
Description: OpenPGP digital signature