On Mon, Jul 29, 2019 at 04:16:28PM +0800, Huang, Ying wrote:
> Srikar Dronamraju <srikar@xxxxxxxxxxxxxxxxxx> writes:
>
> >> >>
> >> >> if (lr_ratio >= NUMA_PERIOD_THRESHOLD)
> >> >> slow down scanning
> >> >> else if (sp_ratio >= NUMA_PERIOD_THRESHOLD) {
> >> >> if (NUMA_PERIOD_SLOTS - lr_ratio >= NUMA_PERIOD_THRESHOLD)
> >> >> speed up scanning
> >>
> >> Thought about this again. For example, a multi-threads workload runs on
> >> a 4-sockets machine, and most memory accesses are shared. The optimal
> >> situation will be pseudo-interleaving, that is, spreading memory
> >> accesses evenly among 4 NUMA nodes. Where "share" >> "private", and
> >> "remote" > "local". And we should slow down scanning to reduce the
> >> overhead.
> >>
> >> What do you think about this?
> >
> > If all 4 nodes have equal access, then all 4 nodes will be active nodes.
> >
> > From task_numa_fault()
> >
> > if (!priv && !local && ng && ng->active_nodes > 1 &&
> > numa_is_active_node(cpu_node, ng) &&
> > numa_is_active_node(mem_node, ng))
> > local = 1;
> >
> > Hence all accesses will be accounted as local. Hence scanning would slow
> > down.
>
> Yes. You are right! Thanks a lot!
>
> There may be another case. For example, a workload with 9 threads runs
> on a 2-sockets machine, and most memory accesses are shared. 7 threads
> runs on the node 0 and 2 threads runs on the node 1 based on CPU load
> balancing. Then the 2 threads on the node 1 will have "share" >>
> "private" and "remote" >> "local". But it doesn't help to speed up
> scanning.
>
Ok, so the results from the patch are mostly neutral. There are some
small differences in scan rates depending on the workload but it's not
universal and the headline performance is sometimes worse. I couldn't
find something that would justify the change on its own. I think in the
short term -- just fix the comments.
For the shared access consideration, the scan rate is important but so too
is the decision on when pseudo interleaving should be used. Both should
probably be taken into account when making changes in this area. The
current code may not be optimal but it also has not generated bug reports,
high CPU usage or obviously bad locality decision in the field. Hence,
for this patch or a similar series, it is critical that some workloads are
selected that really care about the locality of shared access and evaluate
based on that. Initially it was done with a large battery of tests run
by different people but some of those people have changed role since and
would not be in a position to rerun the tests. There also was the issue
that when those were done, NUMA balancing was new so it's comparative
baseline was "do nothing at all".
--
Mel Gorman
SUSE Labs
