On Thu, Jan 13, 2022 at 08:06:40PM +0800, Huang, Ying wrote:
> Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
> > On Thu, Jan 13, 2022 at 03:19:06PM +0800, Huang, Ying wrote:
> >> Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
> >> > On Tue, Dec 07, 2021 at 10:27:51AM +0800, Huang Ying wrote:
> >> >> After commit c221c0b0308f ("device-dax: "Hotplug" persistent memory
> >> >> for use like normal RAM"), the PMEM could be used as the
> >> >> cost-effective volatile memory in separate NUMA nodes. In a typical
> >> >> memory tiering system, there are CPUs, DRAM and PMEM in each physical
> >> >> NUMA node. The CPUs and the DRAM will be put in one logical node,
> >> >> while the PMEM will be put in another (faked) logical node.
> >> >
> >> > So what does a system like that actually look like, SLIT table wise, and
> >> > how does that affect init_numa_topology_type() ?
> >>
> >> The SLIT table is as follows,
<snip>
> >> node distances:
> >> node 0 1 2 3
> >> 0: 10 21 17 28
> >> 1: 21 10 28 17
> >> 2: 17 28 10 28
> >> 3: 28 17 28 10
> >>
> >> init_numa_topology_type() set sched_numa_topology_type to NUMA_DIRECT.
> >>
> >> The node 0 and node 1 are onlined during boot. While the PMEM node,
> >> that is, node 2 and node 3 are onlined later. As in the following dmesg
> >> snippet.
> >
> > But how? sched_init_numa() scans the *whole* SLIT table to determine
> > nr_levels / sched_domains_numa_levels, even offline nodes. Therefore it
> > should find 4 distinct distance values and end up not selecting
> > NUMA_DIRECT.
> >
> > Similarly for the other types it uses for_each_online_node(), which
> > would include the pmem nodes once they've been onlined, but I'm thinking
> > we explicitly want to skip CPU-less nodes in that iteration.
>
> I used the debug patch as below, and get the log in dmesg as follows,
>
> [ 5.394577][ T1] sched_numa_topology_type: 0, levels: 4, max_distance: 28
>
> I found that I forget another caller of init_numa_topology_type() run
> during hotplug. I will add another printk() to show it. Sorry about
> that.
Can you try with this on?
I'm suspecting there's a problem with init_numa_topology_type(); it will
never find the max distance due to the _online_ clause in the iteration,
since you said the pmem nodes are not online yet.
---
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index d201a7052a29..53ab9c63c185 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -1756,6 +1756,8 @@ static void init_numa_topology_type(void)
return;
}
}
+
+ WARN(1, "no NUMA type determined");
}
