On Tue, Jul 30, 2024 at 01:19:53AM -0400, Gregory Price wrote:
> On Tue, Jul 30, 2024 at 09:12:55AM +0800, Huang, Ying wrote:
> > > Right now HMAT appears to be used prescriptively, this despite the fact
> > > that there was a clear intent to separate CPU-nodes and non-CPU-nodes in
> > > the memory-tier code. So this patch simply realizes this intent when the
> > > hints are not very reasonable.
> >
> > If HMAT isn't available, it's hard to put memory devices to
> > appropriate memory tiers without other information. In commit
> > 992bf77591cb ("mm/demotion: add support for explicit memory tiers"),
> > Aneesh pointed out that it doesn't work for his system to put
> > non-CPU-nodes in lower tier.
> >
>
> Per Aneesh in 992bf77591cb - The code explicitly states the intent is
> to put non-CPU-nodes in a lower tier by default.
>
>
> The current implementation puts all nodes with CPU into the highest
> tier, and builds the tier hierarchy by establishing the per-node
> demotion targets based on the distances between nodes.
>
> This is accurate for the current code
>
>
> The current tier initialization code always initializes each
> memory-only NUMA node into a lower tier.
>
> This is *broken* for the currently upstream code.
>
> This appears to be the result of the hmat adistance callback introduction
> (though it may have been broken before that).
>
> ~Gregory
Digging into the history further for the sake of completeness
6c542ab ("mm/demotion: build demotion targets based on ...")
mm/demotion: build demotion targets based on explicit memory tiers
This patch switch the demotion target building logic to use memory
tiers instead of NUMA distance. All N_MEMORY NUMA nodes will be placed
in the default memory tier and additional memory tiers will be added by
drivers like dax kmem.
The decision made in this patch breaks memory-tiers.c for all BIOS
configured CXL devices that generate a DRAM node during early boot,
but for which HMAT is absent or otherwise broken - the new HMAT code
addresses the situation for when HMAT is present.
Hardware supporting this style of configuration has been around for at
least a few years now. I think we should at the very least consider adding
an option to restore this (!N_CPU)=Lower Tier behavior - if not
defaulting to the behavior when HMAT data is not present.
~Gregory
