With Dave Hansen's patches merged into Linus's tree https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c221c0b0308fd01d9fb33a16f64d2fd95f8830a4 PMEM could be hot plugged as NUMA node now. But, how to use PMEM as NUMA node effectively and efficiently is worth exploring. There have been a couple of proposals posted on the mailing list [1] [2] [3]. I already posted two versions of patchset for demoting/promoting memory pages between DRAM and PMEM before this topic was discussed at LSF/MM 2019 (https://lwn.net/Articles/787418/). I do appreciate all the great suggestions from the community. This updated version implemented the most discussion, please see the below design section for the details. Changelog ========= v2 --> v3: * Introduced "migrate mode" for node reclaim. Just do demotion when "migrate mode" is specified per Michal Hocko and Mel Gorman. * Introduced "migrate target" concept for VM per Mel Gorman. The memory nodes which are under DRAM in the hierarchy (i.e. lower bandwidth, higher latency, larger capacity and cheaper than DRAM) are considered as "migrate target" nodes. When "migrate mode" is on, memory reclaim would demote pages to the "migrate target" nodes. * Dropped "twice access" promotion patch per Michal Hocko. * Changed the subject for the patchset to reflect the update. * Rebased to 5.2-rc1. v1 --> v2: * Dropped the default allocation node mask. The memory placement restriction could be achieved by mempolicy or cpuset. * Dropped the new mempolicy since its semantic is not that clear yet. * Dropped PG_Promote flag. * Defined N_CPU_MEM nodemask for the nodes which have both CPU and memory. * Extended page_check_references() to implement "twice access" check for anonymous page in NUMA balancing path. * Reworked the memory demotion code. v2: https://lore.kernel.org/linux-mm/1554955019-29472-1-git-send-email-yang.shi@xxxxxxxxxxxxxxxxx/ v1: https://lore.kernel.org/linux-mm/1553316275-21985-1-git-send-email-yang.shi@xxxxxxxxxxxxxxxxx/ Design ====== With the development of new memory technology, we could have cheaper and larger memory device on the system, which may have higher latency and lower bandwidth than DRAM, i.e. PMEM. It could be used as persistent storage or volatile memory. It fits into the memory hierarchy as a second tier memory. The patchset tries to explore an approach to utilize such memory to improve the memory placement. Basically, the patchset tries to achieve this goal by doing memory promotion/demotion via NUMA balancing and memory reclaim. Introduce a new "migrate" mode for node reclaim. When DRAM has memory pressure, demote pages to PMEM via node reclaim path if "migrate" mode is on. Then NUMA balancing will promote pages to DRAM as long as the page is referenced again. The memory pressure on PMEM node would push the inactive pages of PMEM to disk via swap. Introduce "primary" node and "migrate target" node concepts for VM (patch 1/9 and 2/9). The "primary" node is the node which has both CPU and memory. The "migrate target" node is cpuless node and under DRAM in memory hierarchy (i.e. PMEM may be a suitable one, which has lower bandwidth, higher latency, larger capacity and is cheaper than DRAM). The firmware is effectively going to enforce "cpu-less" nodes for any memory range that has differentiated performance from the conventional memory pool, or differentiated performance for a specific initiator. Defined "N_CPU_MEM" nodemask for the "primary" nodes in order to distinguish with cpuless nodes (memory only, i.e. PMEM nodes) and memoryless nodes (some architectures, i.e. Power, may have memoryless nodes). It is a little bit hard to find out suitable "migrate target" node since this needs firmware exposes the physical characteristics of the memory devices. I'm not quite sure what should be the best way and if it is ready to use now or not. Since PMEM is the only available such device for now, so it sounds retrieving the information from SRAT is the easiest way. We may figure out a better way in the future. The promotion/demotion happens only between "primary" nodes and "migrate target" nodes. No promotion/demotion between "migrate target" nodes and promotion from "primary" nodes to "migrate target" nodes and demotion from "primary" nodes to "migrate target" nodes. This guarantees there is no cycles for memory demotion or promotion. According to the discussion at LFS/MM 2019, "there should only be one node to which pages could be migrated". So reclaim code just tries to demote the pages to the closest "migrate target" node and only tries once. Otherwise "if all nodes in the system were on a fallback list, a page would have to move through every possible option - each RAM-based node and each persistent-memory node - before actually being reclaimed. It would be necessary to maintain the history of where each page has been, and would be likely to disrupt other workloads on the system". This is what v2 patchset does, so keep doing it in the same way in v3. The demotion code moves all the migration candidate pages into one single list, then migrate them together (including THP). This would improve the efficiency of migration according to Zi Yan's research. If the migration fails, the unmigrated pages will be put back to LRU. Use the most opotimistic GFP flags to allocate pages on the "migrate target" node. To reduce the failure rate of demotion, check if the "migrate target" node is contended or not. If the "migrate target" node is contended, just do swap instead of migrate. If migration is failed due to -ENOMEM, mark the node as contended. The contended flag will be cleared once the node get balanced. For now "migrate" mode is not compatible with cpuset and mempolicy since it is hard to get the process's task_struct from struct page. The cpuset and process's mempolicy are stored in task_struct instead of mm_struct. Anonymous page only for the time being since NUMA balancing can't promote unmapped page cache. Page cache can be demoted easily, but promotion is a question, may do it via mark_page_accessed(). Added vmstat counters for pgdemote_kswapd, pgdemote_direct and numa_pages_promoted. There are definitely still a lot of details need to be sorted out. Any comment is welcome. Test ==== The stress test was done with mmtests + applications workload (i.e. sysbench, grep, etc). Generate memory pressure by running mmtest's usemem-stress-numa-compact, then run other applications as workload to stress the promotion and demotion path. The machine was still alive after the stress test had been running for ~30 hours. The /proc/vmstat also shows: ... pgdemote_kswapd 3316563 pgdemote_direct 1930721 ... numa_pages_promoted 81838 [1]: https://lore.kernel.org/linux-mm/20181226131446.330864849@xxxxxxxxx/ [2]: https://lore.kernel.org/linux-mm/20190321200157.29678-1-keith.busch@xxxxxxxxx/ [3]: https://lore.kernel.org/linux-mm/20190404071312.GD12864@xxxxxxxxxxxxxx/T/#me1c1ed102741ba945c57071de9749e16a76e9f3d Yang Shi (9): mm: define N_CPU_MEM node states mm: Introduce migrate target nodemask mm: page_alloc: make find_next_best_node find return migration target node mm: migrate: make migrate_pages() return nr_succeeded mm: vmscan: demote anon DRAM pages to migration target node mm: vmscan: don't demote for memcg reclaim mm: vmscan: check if the demote target node is contended or not mm: vmscan: add page demotion counter mm: numa: add page promotion counter Documentation/sysctl/vm.txt | 6 +++ drivers/acpi/numa.c | 12 +++++ drivers/base/node.c | 4 ++ include/linux/gfp.h | 12 +++++ include/linux/migrate.h | 6 ++- include/linux/mmzone.h | 3 ++ include/linux/nodemask.h | 4 +- include/linux/vm_event_item.h | 3 ++ include/linux/vmstat.h | 1 + include/trace/events/migrate.h | 3 +- mm/compaction.c | 3 +- mm/debug.c | 1 + mm/gup.c | 4 +- mm/huge_memory.c | 4 ++ mm/internal.h | 23 ++++++++ mm/memory-failure.c | 7 ++- mm/memory.c | 4 ++ mm/memory_hotplug.c | 10 +++- mm/mempolicy.c | 7 ++- mm/migrate.c | 33 ++++++++---- mm/page_alloc.c | 20 +++++-- mm/vmscan.c | 186 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- mm/vmstat.c | 14 ++++- 23 files changed, 323 insertions(+), 47 deletions(-)
