On 20.07.23 00:48, Ross Zwisler wrote:
On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote:
On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]
I do think that we need to fix this collision between ZONE_MOVABLE and memmap
allocations, because this issue essentially makes the movablecore= kernel
command line parameter useless in many cases, as the ZONE_MOVABLE region it
creates will often actually be unmovable.
movablecore is kinda hack and I would be more inclined to get rid of it
rather than build more into it. Could you be more specific about your
use case?
The problem that I'm trying to solve is that I'd like to be able to get kernel
core dumps off machines (chromebooks) so that we can debug crashes. Because
the memory used by the crash kernel ("crashkernel=" kernel command line
option) is consumed the entire time the machine is booted, there is a strong
motivation to keep the crash kernel as small and as simple as possible. To
this end I'm trying to get away without SSD drivers, not having to worry about
encryption on the SSDs, etc.
Okay, so you intend to keep the crashkernel area as small as possible.
So, the rough plan right now is:
> 1) During boot set aside some memory that won't contain kernel
allocations.
I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.
We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE
region (or whatever non-kernel region) will be set aside as PMEM in the crash
kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line
parameter passed to the crash kernel.
So, in my sample 4G VM system, I see:
# lsmem --split ZONES --output-all
RANGE SIZE STATE REMOVABLE BLOCK NODE ZONES
0x0000000000000000-0x0000000007ffffff 128M online yes 0 0 None
0x0000000008000000-0x00000000bfffffff 2.9G online yes 1-23 0 DMA32
0x0000000100000000-0x000000012fffffff 768M online yes 32-37 0 Normal
0x0000000130000000-0x000000013fffffff 256M online yes 38-39 0 Movable
Memory block size: 128M
Total online memory: 4G
Total offline memory: 0B
so I'll pass "memmap=256M!0x130000000" to the crash kernel.
2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set
aside only contains user data, which we don't want to store anyway.
I raised that in different context already, but such assumptions are not
100% future proof IMHO. For example, we might at one point be able to
make user page tables movable and place them on there.
But yes, most kernel data structures (which you care about) will
probably never be movable and never end up on these regions.
We make a
filesystem in there, and create a kernel crash dump using 'makedumpfile':
mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump
We then set up the next full kernel boot to also have this same PMEM region,
using the same memmap kernel parameter. We reboot back into a full kernel.
3) The next full kernel will be a normal boot with a full networking stack,
SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out
the kdump and either store it somewhere persistent or upload it somewhere. We
can then unmount the PMEM and reconfigure it back to system ram so that the
live system isn't missing memory.
ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
daxctl reconfigure-device --mode=system-ram dax0.0
This is the flow I'm trying to support, and have mostly working in a VM,
except up until now makedumpfile would crash because all the memblock
structures it needed were in the PMEM area that I had just wiped out by making
a new filesystem. :)
Thinking out loud (and remembering that some architectures relocate the
crashkernel during kexec, if I am not wrong), maybe the following would
also work and make your setup eventually easier:
1) Don't reserve a crashkernel area in the traditional way, instead
reserve that area using CMA. It can be used for MOVABLE allocations.
2) Let kexec load the crashkernel+initrd into ordinary memory only
(consuming as much as you would need there).
