Re: Brainstorming about kernel updates with low disk space

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Sep 7, 2018 at 10:27 AM, Jason L Tibbitts III <tibbs@xxxxxxxxxxx> wrote:
>>>>>> "CM" == Chris Murphy <lists@xxxxxxxxxxxxxxxxx> writes:
>
> CM> A few releases ago, Anaconda now defaults to a 1GiB boot partition
> CM> which should prevent the problem you're describing for anyone using
> CM> the default 3 retained kernels, as well as the extra kdump initramfs
> CM> if you're using kdump (not enabled by default on Fedora).
>
> That is not true; the issue is not about free space in /boot.

OK so the enospc is happening on / not on /boot - I must've blown by
that detail somehow.

Why is /var/tmp running out of space? Is this on a separate /var or /
volume that's just too small from the start, or is /var/tmp
accumulating too much? From 'man systemd-tmpfiles' I can't tell
exactly what "age parameter configured" for the --clean option means.
But it sounds like /var/tmp isn't cleaned up more aggressively when
free space is low, it only does it based on some aging of files.


>The
> kernel package is properly constructed to preserve 20MB in boot and RPM
> itself will not allow the transaction to proceed if that space isn't
> available.  (I would argue that 20MB is cutting it a bit close, but in
> the specific case I had (with a 19MB initramfs file) it was sufficient.

All of my hostonly initramfs files are 22+MB, and the nohostonly is 67+MB.


>
> The specific issue that caused the initramfs to fail to generate was the
> lack free space in /var/tmp for dracut's temporary files.


Gotcha. So why is there a lack of free space there? Sounds like the
volume is too small from the outset, or something is running away and
filling up that volume.

Seems to me the install script should check for /var/tmp free space
before starting. Between kernel and the non-compressed initramfs
creation, it's gonna need maybe upwards of 100M or maybe 150M if these
are nohostonly initramfs's? So test for a minimum of 200M free space,
and if not true, fail?


>
> CM> I think it's in new-kernel-pkg where grubby ends up being called
> CM> twice, and why you see this.
>
> OK, so 20-grub.install will call new-kernel-package.  But the problem
> remains the same: initramfs creation fails, but grub is still updated.
> So maybe the issue isn't in the ordering of the kernel-install scripts
> but in new-kernel-package.

For sure grubby is called twice. And for sure the first time it's
called, the grub.cfg is modified such that a new entry is created
without an initramfs. And to me it's a suboptimal design but appears
to be happening by design. I don't know if it's a grubby limitation
that it works like this, or other.

>
> CM> And modifying grub.cfg is going away. It was supposed to happen for
> CM> Fedora 29, but is being pushed to Fedora 30.
>
> OK, that's good.  It will still be important that whatever configuration
> file actually sets up the bootloader for the new kernel is not enabled
> until after the initramfs has been generated.

I'm not fully understanding the whole sequence of the new method; but
I think the base bootloader configuration script is part of the kernel
package, and is copied from /lib/modules/<kernel>/bls.conf and might
get minimally modified by some kernel script, maybe new-kernel-pkg I'm
not sure, and then dropped into /boot/loader/entries where GRUB later
picks it up. So I agree, the way it ought to work is to not drop that
kernel's conf file into /boot/loader/entries until everything else is
done. The order matters so hopefully it's all happening serially and
not in parallel.



>
> CM> So you could just give up and focus on the new thing being made
> CM> robust if you want.
>
> Seems unfortunate to just ignore the problem.  Maybe it's simply rare
> enough, but to be fair I have seen an unpleasant number of desktops I
> manage not able to reboot without human intervention due to missing
> initramfs files.  I would wager that they're down to the same base issue
> (bootloader config done regardless of whether the initramfs was
> generated) but they aren't all due to lack of free space in /var/tmp.

There's more than one bug here, arguably. Top two bugs in my view
would be not testing for enough free space right from the start. And
then whatever is modifying grubenv before the initramfs and grub.cfg
are for sure committed to stable media.


-- 
Chris Murphy
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux