On Wed, Jun 22, 2022 at 2:04 PM Aneesh Kumar K.V <aneesh.kumar@xxxxxxxxxxxxx> wrote: > > From: Jagdish Gediya <jvgediya@xxxxxxxxxxxxx> > > All N_MEMORY nodes are divided into 3 memoty tiers with tier ID value s /memoty/ memory > MEMORY_TIER_HBM_GPU, MEMORY_TIER_DRAM and MEMORY_TIER_PMEM. By default, > all nodes are assigned to default memory tier. I think adding the default memory tier name will be helpful. > > Demotion path for all N_MEMORY nodes is prepared based on the tier ID value > of memory tiers. > > This patch adds documention for memory tiering introduction, its sysfs > interfaces and how demotion is performed based on memory tiers. > > Suggested-by: Wei Xu <weixugc@xxxxxxxxxx> > Signed-off-by: Jagdish Gediya <jvgediya@xxxxxxxxxxxxx> > Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@xxxxxxxxxxxxx> > --- > Documentation/admin-guide/mm/index.rst | 1 + > .../admin-guide/mm/memory-tiering.rst | 182 ++++++++++++++++++ > 2 files changed, 183 insertions(+) > create mode 100644 Documentation/admin-guide/mm/memory-tiering.rst > > diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst > index c21b5823f126..3f211cbca8c3 100644 > --- a/Documentation/admin-guide/mm/index.rst > +++ b/Documentation/admin-guide/mm/index.rst > @@ -32,6 +32,7 @@ the Linux memory management. > idle_page_tracking > ksm > memory-hotplug > + memory-tiering > nommu-mmap > numa_memory_policy > numaperf > diff --git a/Documentation/admin-guide/mm/memory-tiering.rst b/Documentation/admin-guide/mm/memory-tiering.rst > new file mode 100644 > index 000000000000..142c36651f5d > --- /dev/null > +++ b/Documentation/admin-guide/mm/memory-tiering.rst > @@ -0,0 +1,182 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +.. _admin_guide_memory_tiering: > + > +=========== > +Memory tiers > +============ > + > +This document describes explicit memory tiering support along with > +demotion based on memory tiers. > + > +Introduction > +============ > + > +Many systems have multiple types of memory devices e.g. GPU, DRAM and > +PMEM. The memory subsystem of these systems can be called a memory > +tiering system because the performance of the different types of > +memory is different. Memory tiers are defined based on the hardware > +capabilities of memory nodes. Each memory tier is assigned a tier ID > +value that determines the memory tier position in demotion order. > + > +The memory tier assignment of each node is independent of each > +other. Moving a node from one tier to another tier doesn't affect > +the tier assignment of any other node. > + > +Memory tiers are used to build the demotion targets for nodes. A node > +can demote its pages to any node of any lower tiers. > + > +Memory tier rank > +================= > + > +Memory nodes are divided into 3 types of memory tiers with tier ID > +value as shown based on their hardware characteristics. > + > + > +MEMORY_TIER_HBM_GPU > +MEMORY_TIER_DRAM > +MEMORY_TIER_PMEM > + > +Memory tiers initialization and (re)assignments > +=============================================== > + > +By default, all nodes are assigned to the memory tier with the default tier ID > +DEFAULT_MEMORY_TIER which is 200 (MEMORY_TIER_DRAM). The memory tier of > +the memory node can be either modified through sysfs or from the driver. On > +hotplug, the memory tier with default tier ID is assigned to the memory node. > + > + > +Sysfs interfaces > +================ > + > +Nodes belonging to specific tier can be read from, > +/sys/devices/system/memtier/memtierN/nodelist (Read-Only) > + > +Where N is 0 - 2. > + > +Example 1: > +For a system where Node 0 is CPU + DRAM nodes, Node 1 is HBM node, > +node 2 is a PMEM node an ideal tier layout will be > + > +$ cat /sys/devices/system/memtier/memtier0/nodelist > +1 > +$ cat /sys/devices/system/memtier/memtier1/nodelist > +0 > +$ cat /sys/devices/system/memtier/memtier2/nodelist > +2 > + > +Example 2: > +For a system where Node 0 & 1 are CPU + DRAM nodes, node 2 & 3 are PMEM > +nodes. > + > +$ cat /sys/devices/system/memtier/memtier0/nodelist > +cat: /sys/devices/system/memtier/memtier0/nodelist: No such file or > +directory > +$ cat /sys/devices/system/memtier/memtier1/nodelist > +0-1 > +$ cat /sys/devices/system/memtier/memtier2/nodelist > +2-3 > + > +Default memory tier can be read from, > +/sys/devices/system/memtier/default_tier (Read-Only) > + > +e.g. > +$ cat /sys/devices/system/memtier/default_tier > +memtier200 > + > +Max memory tier ID supported can be read from, > +/sys/devices/system/memtier/max_tier (Read-Only) > + > +e.g. > +$ cat /sys/devices/system/memtier/max_tier > +400 > + > +Individual node's memory tier can be read of set using, > +/sys/devices/system/node/nodeN/memtier (Read-Write) > + > +where N = node id > + > +When this interface is written, Node is moved from the old memory tier > +to new memory tier and demotion targets for all N_MEMORY nodes are > +built again. > + > +For example 1 mentioned above, > +$ cat /sys/devices/system/node/node0/memtier > +1 > +$ cat /sys/devices/system/node/node1/memtier > +0 > +$ cat /sys/devices/system/node/node2/memtier > +2 > + > +Additional memory tiers can be created by writing a tier ID value to this file. > +This results in a new memory tier creation and moving the specific NUMA node to > +that memory tier. > + > +Demotion > +======== > + > +In a system with DRAM and persistent memory, once DRAM > +fills up, reclaim will start and some of the DRAM contents will be > +thrown out even if there is a space in persistent memory. > +Consequently, allocations will, at some point, start falling over to the slower > +persistent memory. > + > +That has two nasty properties. First, the newer allocations can end up in > +the slower persistent memory. Second, reclaimed data in DRAM are just > +discarded even if there are gobs of space in persistent memory that could > +be used. > + > +Instead of a page being discarded during reclaim, it can be moved to > +persistent memory. Allowing page migration during reclaim enables > +these systems to migrate pages from fast(higher) tiers to slow(lower) > +tiers when the fast(higher) tier is under pressure. > + > + > +Enable/Disable demotion > +----------------------- > + > +By default demotion is disabled, it can be enabled/disabled using > +below sysfs interface, > + > +$ echo 0/1 or false/true > /sys/kernel/mm/numa/demotion_enabled > + > +preferred and allowed demotion nodes > +------------------------------------ > + > +Preferred nodes for a specific N_MEMORY node are the best nodes > +from the next possible lower memory tier. Allowed nodes for any > +node are all the nodes available in all possible lower memory > +tiers. > + > +Example: > + > +For a system where Node 0 & 1 are CPU + DRAM nodes, node 2 & 3 are PMEM > +nodes, > + > +node distances: > +node 0 1 2 3 > + 0 10 20 30 40 > + 1 20 10 40 30 > + 2 30 40 10 40 > + 3 40 30 40 10 > + > +memory_tiers[0] = <empty> > +memory_tiers[1] = 0-1 > +memory_tiers[2] = 2-3 > + > +node_demotion[0].preferred = 2 > +node_demotion[0].allowed = 2, 3 > +node_demotion[1].preferred = 3 > +node_demotion[1].allowed = 3, 2 > +node_demotion[2].preferred = <empty> > +node_demotion[2].allowed = <empty> > +node_demotion[3].preferred = <empty> > +node_demotion[3].allowed = <empty> > + > +Memory allocation for demotion > +------------------------------ > + > +If a page needs to be demoted from any node, the kernel 1st tries > +to allocate a new page from the node's preferred node and fallbacks to > +node's allowed targets in allocation fallback order. > + > -- > 2.36.1 > >
