>On 24.03.23 10:27, Kyungsan Kim wrote: >>> On 24.03.23 10:09, Kyungsan Kim wrote: >>>> Thank you David Hinderbrand for your interest on this topic. >>>> >>>>>> >>>>>>> Kyungsan Kim wrote: >>>>>>> [..] >>>>>>>>> In addition to CXL memory, we may have other kind of memory in the >>>>>>>>> system, for example, HBM (High Bandwidth Memory), memory in FPGA card, >>>>>>>>> memory in GPU card, etc. I guess that we need to consider them >>>>>>>>> together. Do we need to add one zone type for each kind of memory? >>>>>>>> >>>>>>>> We also don't think a new zone is needed for every single memory >>>>>>>> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to >>>>>>>> manage multiple volatile memory devices due to the increased device >>>>>>>> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to >>>>>>>> represent extended volatile memories that have different HW >>>>>>>> characteristics. >>>>>>> >>>>>>> Some advice for the LSF/MM discussion, the rationale will need to be >>>>>>> more than "we think the ZONE_EXMEM can be used to represent extended >>>>>>> volatile memories that have different HW characteristics". It needs to >>>>>>> be along the lines of "yes, to date Linux has been able to describe DDR >>>>>>> with NUMA effects, PMEM with high write overhead, and HBM with improved >>>>>>> bandwidth not necessarily latency, all without adding a new ZONE, but a >>>>>>> new ZONE is absolutely required now to enable use case FOO, or address >>>>>>> unfixable NUMA problem BAR." Without FOO and BAR to discuss the code >>>>>>> maintainability concern of "fewer degress of freedom in the ZONE >>>>>>> dimension" starts to dominate. >>>>>> >>>>>> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. >>>>>> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. >>>>>> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. >>>>>> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. >>>> >>>>> That sounds like a bad hack :) . >>>> I consent you. >>>> >>>>>> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. >>>>>> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. >>>> >>>>> I once raised the idea of a ZONE_PREFER_MOVABLE [1], maybe that's >>>>> similar to what you have in mind here. In general, adding new zones is >>>>> frowned upon. >>>> >>>> Actually, we have already studied your idea and thought it is similar with us in 2 aspects. >>>> 1. ZONE_PREFER_MOVABLE allows a kernelspace allocation using a new zone >>>> 2. ZONE_PREFER_MOVABLE helps less fragmentation by splitting zones, and ordering allocation requests from the zones. >>>> >>>> We think ZONE_EXMEM also helps less fragmentation. >>>> Because it is a separated zone and handles a page allocation as movable by default. >>> >>> So how is it different that it would justify a different (more confusing >>> IMHO) name? :) Of course, names don't matter that much, but I'd be >>> interested in which other aspect that zone would be "special". >> >> FYI for the first time I named it as ZONE_CXLMEM, but we thought it would be needed to cover other extended memory types as well. >> So I changed it as ZONE_EXMEM. >> We also would like to point out a "special" zone aspeact, which is different from ZONE_NORMAL for tranditional DDR DRAM. >> Of course, a symbol naming is important more or less to represent it very nicely, though. >> Do you prefer ZONE_SPECIAL? :) > >I called it ZONE_PREFER_MOVABLE. If you studied that approach there must >be a good reason to name it differently? > The intention of ZONE_EXMEM is a separated logical management dimension originated from the HW diffrences of extended memory devices. Althought the ZONE_EXMEM considers the movable and frementation aspect, it is not all what ZONE_EXMEM considers. So it is named as it.
