On 23.03.23 11:51, Kyungsan Kim wrote:
I appreciate dan for the careful advice.
Kyungsan Kim wrote:
[..]
In addition to CXL memory, we may have other kind of memory in the
system, for example, HBM (High Bandwidth Memory), memory in FPGA card,
memory in GPU card, etc. I guess that we need to consider them
together. Do we need to add one zone type for each kind of memory?
We also don't think a new zone is needed for every single memory
device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to
manage multiple volatile memory devices due to the increased device
types. Including CXL DRAM, we think the ZONE_EXMEM can be used to
represent extended volatile memories that have different HW
characteristics.
Some advice for the LSF/MM discussion, the rationale will need to be
more than "we think the ZONE_EXMEM can be used to represent extended
volatile memories that have different HW characteristics". It needs to
be along the lines of "yes, to date Linux has been able to describe DDR
with NUMA effects, PMEM with high write overhead, and HBM with improved
bandwidth not necessarily latency, all without adding a new ZONE, but a
new ZONE is absolutely required now to enable use case FOO, or address
unfixable NUMA problem BAR." Without FOO and BAR to discuss the code
maintainability concern of "fewer degress of freedom in the ZONE
dimension" starts to dominate.
One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases.
ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time.
ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation.
Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag.
That sounds like a bad hack :) .
In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped.
We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases.
I once raised the idea of a ZONE_PREFER_MOVABLE [1], maybe that's
similar to what you have in mind here. In general, adding new zones is
frowned upon.
As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity.
So, we thought it could be a graceful approach adding a new zone and separately manage the new features.
Kindly let me know any advice or comment on our thoughts.
[1] https://www.lkml.org/lkml/2020/9/9/667
--
Thanks,
David / dhildenb
