I'd like to brainstorm a bit about how to make kernel updates more
resilient to running out of disk space. I've talked to a few people on
IRC about this but I'm not at the point where I know enough to file good
bugs and figured that a larger discussion might help.
Summary of my questions:
Why is the grub configuration modified to make a newly installed kernel bootable
before the initramfs has been generated?
Why do errors from generating the initramfs not have any effect on the
kernel installation process?
Is there anything (other than perhaps fixing the above two weirdnesses)
that we can reasonably do to prevent systems from ending up in the state
mine were in? It's easy (but slightly messy) to make the kernel package
require some free space in /var/tmp; is it reasonable to do that?
Longer discussion:
The other day I found a couple of VMs wouldn't reboot because the
initramfs files were missing. (I could still boot an older kernel,
fortunately.) In turn, that was due to the filesystem holding /var/tmp
being quite full.
I did some digging and discovered the following:
The kernel package is set up properly to ensure that the RPM will fail
to install if there isn't space for the generated initramfs file in
/boot. (That file is marked %ghost, but the kernel spec creates a 20MB
dummy file so that RPM will ensure the space is available.)
After the kernel package is installed, its %posttrans scriptlet is run
which just calls /bin/kernel-install. A failure here is recorded in the
dnf history and logged in /var/log/dnf.rpm.log:
2018-09-02T08:23:44Z ERROR Non-fatal POSTTRANS scriptlet failure in rpm package kernel-core
The kernel-install script is part of the systemd-udev package. It just
runs scripts in order from /{etc,usr/lib}/kernel/install.d. It keeps
track of errors from these scripts but does not act on them, except for
the magic return code of 77, which causes it to stop processing scripts.
If one or more of the scripts fails, install-kernel will exit with the
sum of their error codes, but a failure here can't do anything like
uninstall the problematic package.
On my systems, /etc/kernel/install.d/20-grubby.install and
90-loaderentry.install are empty. I believe this means that they mask
the scripts with the same names in /usr/lib/kernel/install.d. This
means the following scripts are run on kernel install, in this order:
* 20-grub.install
* 50-depmod.install
* 50-dracut.install
* 51-dracut-rescue.install
And running dracut is another big bunch of scripts and plugins; I didn't
dig down that far. However, it does appear that dracut will compose the
initramfs in /var/tmp, compress it and move it into place. This means
that it requires something on the order of twice the initramfs size in
order to run.
So, my questions:
* Why is the grub configuration changed first, before dracut gets to
run? If dracut fails, the system is probably not going to be able to
boot that kernel, so setting it up as the default as the first action
taken doesn't seem like a great idea.
* Why are errors in these scripts ignored by kernel-install? Even if
grub were moved later in the process, it would still be run even if an
earlier script failed.
* If those two were fixed, the end result of a dracut failure would be
that the kernel packages are installed but don't show up in the grub
menu. Is there anything we can reasonably do about that?
* The kernel could "pre-allocate" space in /var/tmp (by %ghosting a
dummy nonzero-length file there) so that the package simply won't
install if there isn't space sufficient to generate the initramfs.
Would that be something we could reasonably do?
This would result in the kernel package "owning" a file in /var/tmp
that doesn't exist. The name doesn't actually matter and the file
wouldn't actually be in the package. RPM would just ensure that
enough space would exist in /var/tmp before allowing the installation
to begin.
Of course, this requires the kernel package to guess as the size of
the initramfs, and to begin to care about how much space dracut needs
and where it stores its temporary files. It already guesses about the
size of the initramfs (20MB, which is just larger than the smallest I
can get).
- J<
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
