Thank you David Hinderbrand for your interest on this topic. >> >>> Kyungsan Kim wrote: >>> [..] >>>>> In addition to CXL memory, we may have other kind of memory in the >>>>> system, for example, HBM (High Bandwidth Memory), memory in FPGA card, >>>>> memory in GPU card, etc. I guess that we need to consider them >>>>> together. Do we need to add one zone type for each kind of memory? >>>> >>>> We also don't think a new zone is needed for every single memory >>>> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to >>>> manage multiple volatile memory devices due to the increased device >>>> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to >>>> represent extended volatile memories that have different HW >>>> characteristics. >>> >>> Some advice for the LSF/MM discussion, the rationale will need to be >>> more than "we think the ZONE_EXMEM can be used to represent extended >>> volatile memories that have different HW characteristics". It needs to >>> be along the lines of "yes, to date Linux has been able to describe DDR >>> with NUMA effects, PMEM with high write overhead, and HBM with improved >>> bandwidth not necessarily latency, all without adding a new ZONE, but a >>> new ZONE is absolutely required now to enable use case FOO, or address >>> unfixable NUMA problem BAR." Without FOO and BAR to discuss the code >>> maintainability concern of "fewer degress of freedom in the ZONE >>> dimension" starts to dominate. >> >> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. >> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. >> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. >> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. >That sounds like a bad hack :) . I consent you. >> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. >> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. >I once raised the idea of a ZONE_PREFER_MOVABLE [1], maybe that's >similar to what you have in mind here. In general, adding new zones is >frowned upon. Actually, we have already studied your idea and thought it is similar with us in 2 aspects. 1. ZONE_PREFER_MOVABLE allows a kernelspace allocation using a new zone 2. ZONE_PREFER_MOVABLE helps less fragmentation by splitting zones, and ordering allocation requests from the zones. We think ZONE_EXMEM also helps less fragmentation. Because it is a separated zone and handles a page allocation as movable by default. >> As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity. >> So, we thought it could be a graceful approach adding a new zone and separately manage the new features. >> >> Kindly let me know any advice or comment on our thoughts. > >[1] https://www.lkml.org/lkml/2020/9/9/667 > >-- >Thanks, > >David / dhildenb
