On Wed, Apr 14, 2021 at 1:08 AM Oscar Salvador <osalvador@xxxxxxx> wrote: > > Hi Wei Xu, > > I have some questions about it > > Fast class/memory are pictured as those nodes with CPUs, while Slow class/memory > are PMEM, right? > Then, what stands for medium class/memory? That is Dave's example. I think David's guess makes sense (HBM - fast, DRAM - medium, PMEM - slow). It may also be possible that we have DDR5 as fast, CXL-DDR4 as medium, and CXL-PMEM as slow. But the most likely use cases for now should be just two tiers: DRAM vs PMEM or other types of slower memory devices. > In Dave's example, list is created in a way that stays local to the socket, > and we go from the fast one to the slow one. > In yours, lists are created taking the fastest nodes from all sockets and > we work our way down, which means have cross-socket nodes in the list. > How much of a penalty is that? Cross-socket demotion is certainly more expensive. But because it is sequential access and can also be optimized with non-temporal stores, it may not be much slower than demotion to a local node in the next tier. The actual penalty will depend on the devices. > And while I get your point, I am not sure if that is what we pretend here. > This patchset aims to place cold pages that are about to be reclaim in slower > nodes to give them a second chance, while your design seems more to have kind > of different memory clases and be able to place applications in one of those tiers > depending on its demands or sysadmin-demand. > > Could you expand some more? Sure. What I have described has the same goal as Dave's patchset, i,e, to demote cold pages to the slower nodes when they are about to be reclaimed. The only difference is that in my suggestion the demotion target of a fast tier node is expanded from a single node to a set of nodes from the slow tier and one node in such a set can be marked as the preferred/local demotion target. This can help enable more flexible demotion policies to be configured, such as to allow a cgroup to allocate from all fast tier nodes, but only demote to a local slow tier node. Such a policy can reduce memory stranding at the fast tier (compared to if memory hardwall is used) and still allow demotion from all fast tier nodes without incurring the expensive random accesses to the demoted pages if they were demoted to remote slow tier nodes. I understand that Dave started this patchset with a simplified demotion path definition, which I agree. Meanwhile, I think this more generalized definition of demotion path is useful and can also be important for some use cases.
