On 3/26/23 02:21, Mike Rapoport wrote:
Hi,
[..] >> One problem we experienced was occured in the combination of
hot-remove and kerelspace allocation usecases.
ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time.
ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation.
Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag.
In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped.
We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases.
As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity.
So, we thought it could be a graceful approach adding a new zone and separately manage the new features.
This still does not describe what are the use cases that require having
kernel allocations on CXL.mem.
I believe it's important to start with explanation *why* it is important to
have kernel allocations on removable devices.
Hi Mike,
not speaking for Kyungsan here, but I am starting to tackle hypervisor
clustering and VM migration over cxl.mem [1].
And in my mind, at least one reason that I can think of having kernel
allocations from cxl.mem devices is where you have multiple VH
connections sharing the memory [2]. Where for example you have a user
space application stored in cxl.mem, and then you want the metadata
about this process/application that the kernel keeps on one hypervisor
be "passed on" to another hypervisor. So basically the same way
processors in a single hypervisors cooperate on memory, you extend that
across processors that span over physical hypervisors. If that makes
sense...
[1] A high-level explanation is at http://nil-migration.org
[2] Compute Express Link Specification r3.0, v1.0 8/1/22, Page 51,
figure 1-4, black color scheme circle(3) and bars.
--
Peace can only come as a natural consequence
of universal enlightenment -Dr. Nikola Tesla
