>> While there are a couple of ideas floating around here, my current
>> suggestion would be either
>>
>> 1. Indicate all hotplugged memory as "System RAM (hotplugged)" in
>> /proc/iomem and the firmware memmap (on all architectures). This will
>> require kexec changes,
>
>> but I would have assume that kexec has to be
>> updated in lock-step with the kernel
>
> News to me: I was using the version I first built when arm64's support was new. I've only
> had to update it once when we had to change user-space.
>
> I don't think debian updates kexec-tools when it updates the kernel.
I would assume they are also not pushing the latest-greatest kernel in
their current release, after settling on a kexec version, no?
I think you can assume new kernels to require new kexec-tools versions
to provide all features.
> Making changes to /proc/iomem means updating user-space again, (for kdump). I'd like to
> avoid that if its at all possible.
Yes, it's not desirable, but if all that's not working is a "not all
memory will be dumped out of the box", at least I think this is
tolerable. It's not like we're completely breaking kexec.
Your current arm64 patches require the same change AFAIKS - and I think
we already have arm64 hotplug support in Linux distros.
As I said, similarly, makedumpfile has to be upgraded with every kernel
release to make kdump work as expected. And that is no big news I hope :)
>> just like e.g., makedumpfile.
>> Modify kexec() to not place the kexec kernel on these areas (easy) but
>> still consider them as crash regions to dump. When loading a kexec
>> kernel, validate in the kernel that the memory is appropriate.
>
>
>> 2. Make kexec() reload the the kernel whenever we e.g., get a udev event
>> for removal of memory in /sys/devices/system/memory/.
>
> I don't think we can rely on user-space to do something,
>
>
>> On every remove_memory(), invalidate the loaded kernel in the kernel.
>
> This is an option, ... but its a change of behaviour. If user-space asks for two
> impossible things, the second request should fail. Having the first-one disappear is a bit
> spooky...
We are talking about corner cases that are already broken, no?
>
> Fortunately user-space checks the 'kexec -l' bit happened before it calls reboot() behind
> 'kexec -e'. So this works, but is not intuitive.
>
> ("Did I load it? What changed and when? oh, half a mile up in dmesg is a message saying
> the kernel discarded the kexec kernel last wednesday.")
>
>
>> As I mentioned somewhere, 1. will be interesting for virtio-mem, where
>> we don't want any kexec kernel to be placed on virtio-mem-added memory.
>
> Do these virtio-mem-added regions need to be accessible by kdump?
> (do we already need a user-space change for that?)
Yes, they have to be accessible by kdump. Currently, they are also
exported as "System RAM" via /proc/iomem - which is why dumping works
e.g., on x86-64 (we'll have to increase the #of memory resources that
can be considered in the future, but that's a different story and only
applies when adding more than 100GB of memory via virtio-mem or so)
But as virtio-mem is fairly new (IOW, about to get queued for
integration soonish), I could still change the memory resources to show
up differently ("System RAM (hotplugged)", "System RAM (virtio-mem)",
etc.) and teach kexec about them.
But learning that we are having similar problems on arm64 (and
theoretically on Hyper-V), I think it makes sense to discuss a solution
that will solve the other issues as well.
>
>
> A third option, along the line of what I posted:
>
> Split the 'offline' and 'removed' ideas, which David mentioned somewhere. We'd end up with
> (yet) another notifier chain, that prevents the memory being removed, but you can still
I dislike limiting memory unplug - and especially making remove_memory()
fail - just because somebody once thought it would be a good place to
load - in the future - some kexec binary onto it.
> mark it as offline in /sys/. (...I'm not quite sure why you would do that...)
>
> This would need hooking up for ACPI (which covers x86 and arm64), and other architectures
> mechanisms for doing this...
> arm64 can then switch is arch hook that prevents 'bootmem' being removed to this new
> notifier chain, as the kernel can only boot from that was present at boot.
We have two different problems here, right?
1. Don't place kexec binaries on specific memory areas (e.g., arm64,
virtio-mem, hyper-v, ...)
2. Figure out what to do when unplugging memory that was selected as a
target for kexec binaries.
For 1, I have a feeling that /proc/iomem could be the right solution,
eventually requiring kexec changes to handle kdump properly (IOW, dump
all memory). Indicating all hotplugged memory as "System RAM
(hotplugged)" would be the way to go here.
For 2, I think we should unload all kexec images in case they overlap
with memory to be removed (e.g., remove_memory() notifier, which cannot
stop removal, it's only an indication), and make userspace reload kexec
via udev events.
Also, we have to think about kexec_file_load() to deal with 1.
--
Thanks,
David / dhildenb
