On Thu, Apr 28, 2022 at 12:30 PM Chen, Tim C <tim.c.chen@xxxxxxxxx> wrote: > > > > >On Wed, 2022-04-27 at 09:27 -0700, Wei Xu wrote: > >> On Wed, Apr 27, 2022 at 12:11 AM ying.huang@xxxxxxxxx > >> <ying.huang@xxxxxxxxx> wrote: > >> > > >> > On Mon, 2022-04-25 at 09:56 -0700, Wei Xu wrote: > >> > > On Sat, Apr 23, 2022 at 8:02 PM ying.huang@xxxxxxxxx > >> > > <ying.huang@xxxxxxxxx> wrote: > >> > > > > >> > > > Hi, All, > >> > > > > >> > > > On Fri, 2022-04-22 at 16:30 +0530, Jagdish Gediya wrote: > >> > > > > >> > > > [snip] > >> > > > > >> > > > > I think it is necessary to either have per node demotion > >> > > > > targets configuration or the user space interface supported by > >> > > > > this patch series. As we don't have clear consensus on how the > >> > > > > user interface should look like, we can defer the per node > >> > > > > demotion target set interface to future until the real need arises. > >> > > > > > >> > > > > Current patch series sets N_DEMOTION_TARGET from dax device > >> > > > > kmem driver, it may be possible that some memory node desired > >> > > > > as demotion target is not detected in the system from dax-device > >kmem probe path. > >> > > > > > >> > > > > It is also possible that some of the dax-devices are not > >> > > > > preferred as demotion target e.g. HBM, for such devices, node > >> > > > > shouldn't be set to N_DEMOTION_TARGETS. In future, Support > >> > > > > should be added to distinguish such dax-devices and not mark > >> > > > > them as N_DEMOTION_TARGETS from the kernel, but for now this > >> > > > > user space interface will be useful to avoid such devices as demotion > >targets. > >> > > > > > >> > > > > We can add read only interface to view per node demotion > >> > > > > targets from /sys/devices/system/node/nodeX/demotion_targets, > >> > > > > remove duplicated /sys/kernel/mm/numa/demotion_target > >> > > > > interface and instead make > >/sys/devices/system/node/demotion_targets writable. > >> > > > > > >> > > > > Huang, Wei, Yang, > >> > > > > What do you suggest? > >> > > > > >> > > > We cannot remove a kernel ABI in practice. So we need to make > >> > > > it right at the first time. Let's try to collect some > >> > > > information for the kernel ABI definitation. > >> > > > > >> > > > The below is just a starting point, please add your requirements. > >> > > > > >> > > > 1. Jagdish has some machines with DRAM only NUMA nodes, but they > >> > > > don't want to use that as the demotion targets. But I don't > >> > > > think this is a issue in practice for now, because > >> > > > demote-in-reclaim is disabled by default. > >> > > > > >> > > > 2. For machines with PMEM installed in only 1 of 2 sockets, for > >> > > > example, > >> > > > > >> > > > Node 0 & 2 are cpu + dram nodes and node 1 are slow memory node > >> > > > near node 0, > >> > > > > >> > > > available: 3 nodes (0-2) > >> > > > node 0 cpus: 0 1 > >> > > > node 0 size: n MB > >> > > > node 0 free: n MB > >> > > > node 1 cpus: > >> > > > node 1 size: n MB > >> > > > node 1 free: n MB > >> > > > node 2 cpus: 2 3 > >> > > > node 2 size: n MB > >> > > > node 2 free: n MB > >> > > > node distances: > >> > > > node 0 1 2 > >> > > > 0: 10 40 20 > >> > > > 1: 40 10 80 > >> > > > 2: 20 80 10 > >> > > > > >> > > > We have 2 choices, > >> > > > > >> > > > a) > >> > > > node demotion targets > >> > > > 0 1 > >> > > > 2 1 > >> > > > > >> > > > b) > >> > > > node demotion targets > >> > > > 0 1 > >> > > > 2 X > >> > > > > >> > > > a) is good to take advantage of PMEM. b) is good to reduce > >> > > > cross-socket traffic. Both are OK as defualt configuration. > >> > > > But some users may prefer the other one. So we need a user > >> > > > space ABI to override the default configuration. > >> > > > >> > > I think 2(a) should be the system-wide configuration and 2(b) can > >> > > be achieved with NUMA mempolicy (which needs to be added to > >demotion). > >> > > >> > Unfortunately, some NUMA mempolicy information isn't available at > >> > demotion time, for example, mempolicy enforced via set_mempolicy() > >> > is for thread. But I think that cpusets can work for demotion. > >> > > >> > > In general, we can view the demotion order in a way similar to > >> > > allocation fallback order (after all, if we don't demote or > >> > > demotion lags behind, the allocations will go to these demotion > >> > > target nodes according to the allocation fallback order anyway). > >> > > If we initialize the demotion order in that way (i.e. every node > >> > > can demote to any node in the next tier, and the priority of the > >> > > target nodes is sorted for each source node), we don't need > >> > > per-node demotion order override from the userspace. What we need > >> > > is to specify what nodes should be in each tier and support NUMA > >mempolicy in demotion. > >> > > >> > This sounds interesting. Tier sounds like a natural and general > >> > concept for these memory types. It's attracting to use it for user > >> > space interface too. For example, we may use that for mem_cgroup > >> > limits of a specific memory type (tier). > >> > > >> > And if we take a look at the N_DEMOTION_TARGETS again from the "tier" > >> > point of view. The nodes are divided to 2 classes via > >> > N_DEMOTION_TARGETS. > >> > > >> > - The nodes without N_DEMOTION_TARGETS are top tier (or tier 0). > >> > > >> > - The nodes with N_DEMOTION_TARGETS are non-top tier (or tier 1, 2, > >> > 3, > >> > ...) > >> > > >> > >> Yes, this is one of the main reasons why we (Google) want this interface. > >> > >> > So, another possibility is to fit N_DEMOTION_TARGETS and its > >> > overriding into "tier" concept too. !N_DEMOTION_TARGETS == TIER0. > >> > > >> > - All nodes start with TIER0 > >> > > >> > - TIER0 can be cleared for some nodes via e.g. kmem driver > >> > > >> > TIER0 node list can be read or overriden by the user space via the > >> > following interface, > >> > > >> > /sys/devices/system/node/tier0 > >> > > >> > In the future, if we want to customize more tiers, we can add tier1, > >> > tier2, tier3, ..... For now, we can add just tier0. That is, the > >> > interface is extensible in the future compared with > >> > .../node/demote_targets. > >> > > >> > >> This more explicit tier definition interface works, too. > >> > > > >In addition to make tiering definition explicit, more importantly, this makes it > >much easier to support more than 2 tiers. For example, for a system with > >HBM (High Bandwidth Memory), CPU+DRAM, DRAM only, and PMEM, that is, > >3 tiers, we can put HBM in tier 0, CPU+DRAM and DRAM only in tier 1, and > >PMEM in tier 2, automatically, or via user space overridding. > >N_DEMOTION_TARGETS isn't natural to be extended to support this. > > Agree with Ying that making the tier explicit is fundamental to the rest of the API. > > I think that the tier organization should come before setting the demotion targets, > not the other way round. > > That makes things clear on the demotion direction, (node in tier X > demote to tier Y, X<Y). With that, explicitly specifying the demotion target or > order is only needed when we truly want that level of control or a demotion > order. Otherwise all the higher numbered tiers are valid targets. > Configuring a tier level for each node is a lot easier than fixing up all > demotion targets for each and every node. > > We can prevent demotion target configuration that goes in the wrong > direction by looking at the tier level. > > Tim > I have just posted a RFC on the tier-oriented memory tiering kernel interface based on the discussions here. The RFC proposes a sysfs interface, /sys/devices/system/node/memory_tiers, to display and override the nodes in each memory tier. It also proposes that we rely on the kernel allocation order to select the demotion target node from the next tier and don't expose a userspace overriding interface for per-node demotion order. The RFC drops the approach of CPU nodes as the top-tier by default, too.
