On Wed, 1 Jul 2020 12:45:17 -0700
David Rientjes <rientjes@xxxxxxxxxx> wrote:
> On Wed, 1 Jul 2020, Yang Shi wrote:
>
> > > We can do this if we consider pmem not to be a separate memory tier from
> > > the system perspective, however, but rather the socket perspective. In
> > > other words, a node can only demote to a series of exclusive pmem ranges
> > > and promote to the same series of ranges in reverse order. So DRAM node 0
> > > can only demote to PMEM node 2 while DRAM node 1 can only demote to PMEM
> > > node 3 -- a pmem range cannot be demoted to, or promoted from, more than
> > > one DRAM node.
> > >
> > > This naturally takes care of mbind() and cpuset.mems if we consider pmem
> > > just to be slower volatile memory and we don't need to deal with the
> > > latency concerns of cross socket migration. A user page will never be
> > > demoted to a pmem range across the socket and will never be promoted to a
> > > different DRAM node that it doesn't have access to.
> >
> > But I don't see too much benefit to limit the migration target to the
> > so-called *paired* pmem node. IMHO it is fine to migrate to a remote (on a
> > different socket) pmem node since even the cross socket access should be much
> > faster then refault or swap from disk.
> >
>
> Hi Yang,
>
> Right, but any eventual promotion path would allow this to subvert the
> user mempolicy or cpuset.mems if the demoted memory is eventually promoted
> to a DRAM node on its socket. We've discussed not having the ability to
> map from the demoted page to either of these contexts and it becomes more
> difficult for shared memory. We have page_to_nid() and page_zone() so we
> can always find the appropriate demotion or promotion node for a given
> page if there is a 1:1 relationship.
>
> Do we lose anything with the strict 1:1 relationship between DRAM and PMEM
> nodes? It seems much simpler in terms of implementation and is more
> intuitive.
Hi David, Yang,
The 1:1 mapping implies a particular system topology. In the medium
term we are likely to see systems with a central pool of persistent memory
with equal access characteristics from multiple CPU containing nodes, each
with local DRAM.
Clearly we could fake a split of such a pmem pool to keep the 1:1 mapping
but it's certainly not elegant and may be very wasteful for resources.
Can a zone based approach work well without such a hard wall?
Jonathan
>
> > I think using pmem as a node is more natural than zone and less intrusive
> > since we can just reuse all the numa APIs. If we treat pmem as a new zone I
> > think the implementation may be more intrusive and complicated (i.e. need a
> > new gfp flag) and user can't control the memory placement.
> >
>
> This is an important decision to make, I'm not sure that we actually
> *want* all of these NUMA APIs :) If my memory is demoted, I can simply do
> migrate_pages() back to DRAM and cause other memory to be demoted in its
> place. Things like MPOL_INTERLEAVE over nodes {0,1,2} don't make sense.
> Kswapd for a DRAM node putting pressure on a PMEM node for demotion that
> then puts the kswapd for the PMEM node under pressure to reclaim it serves
> *only* to spend unnecessary cpu cycles.
>
> Users could control the memory placement through a new mempolicy flag,
> which I think are needed anyway for explicit allocation policies for PMEM
> nodes. Consider if PMEM is a zone so that it has the natural 1:1
> relationship with DRAM, now your system only has nodes {0,1} as today, no
> new NUMA topology to consider, and a mempolicy flag MPOL_F_TOPTIER that
> specifies memory must be allocated from ZONE_MOVABLE or ZONE_NORMAL (and I
> can then mlock() if I want to disable demotion on memory pressure).
>
