Hi David,
On 22 Oct 2018, at 17:04, David Rientjes wrote:
On Tue, 16 Oct 2018, Mel Gorman wrote:
I consider this to be an unfortunate outcome. On the one hand, we
have a
problem that three people can trivially reproduce with known test
cases
and a patch shown to resolve the problem. Two of those three people
work
on distributions that are exposed to a large number of users. On the
other, we have a problem that requires the system to be in a specific
state and an unknown workload that suffers badly from the remote
access
penalties with a patch that has review concerns and has not been
proven
to resolve the trivial cases.
The specific state is that remote memory is fragmented as well, this
is
not atypical. Removing __GFP_THISNODE to avoid thrashing a zone will
only
be beneficial when you can allocate remotely instead. When you cannot
allocate remotely instead, you've made the problem much worse for
something that should be __GFP_NORETRY in the first place (and was for
years) and should never thrash.
I'm not interested in patches that require remote nodes to have an
abundance of free or unfragmented memory to avoid regressing.
I just wonder what is the page allocation priority list in your
environment,
assuming all memory nodes are so fragmented that no huge pages can be
obtained without compaction or reclaim.
Here is my version of that list, please let me know if it makes sense to
you:
1. local huge pages: with compaction and/or page reclaim, you are
willing
to pay the penalty of getting huge pages;
2. local base pages: since, in your system, remote data accesses have
much
higher penalty than the extra TLB misses incurred by the base page size;
3. remote huge pages: at least it is better than remote base pages;
4. remote base pages: it performs worst in terms of locality and TLBs.
This might not be easy to implement in current kernel, because
the zones from remote nodes will always be candidates when
kernel is trying get_page_from_freelist(). Only _GFP_THISNODE
and MPOL_BIND can eliminate these remote node zones, where _GFP_THISNODE
is a kernel version MPOL_BIND and overwrites any user space
memory policy other than MPOL_BIND, which is troublesome.
In addition, to prioritize local base pages over remote pages,
the original huge page allocation has to fail, then kernel can
fall back to base page allocations. And you will never get remote
huge pages any more if the local base page allocation fails,
because there is no way back to huge page allocation after the fallback.
Do you expect both behaviors?
In the case of distributions, the first
patch addresses concerns with a common workload where on the other
hand
we have an internal workload of a single company that is affected --
which indirectly affects many users admittedly but only one entity
directly.
The alternative, which is my patch, hasn't been tested or shown why it
cannot work. We continue to talk about order >= pageblock_order vs
__GFP_COMPACTONLY.
I'd like to know, specifically:
- what measurable affect my patch has that is better solved with
removing
__GFP_THISNODE on systems where remote memory is also fragmented?
- what platforms benefit from remote access to hugepages vs accessing
local small pages (I've asked this maybe 4 or 5 times now)?
- how is reclaiming (and possibly thrashing) memory helpful if
compaction
fails to free an entire pageblock due to slab fragmentation due to
low
on memory conditions and the page allocator preference to return
node-
local memory?
- how is reclaiming (and possibly thrashing) memory helpful if
compaction
cannot access the memory reclaimed because the freeing scanner has
already passed by it, or the migration scanner has passed by it,
since
this reclaim is not targeted to pages it can find?
- what metrics can be introduced to the page allocator so that we can
determine that reclaiming (and possibly thrashing) memory will
result
in a hugepage being allocated?
The slab fragmentation and whether reclaim/compaction can help form
huge pages seem to orthogonal to this patch, which tries to decide
the priority between locality and huge pages.
For slab fragmentation, you might find this paper “Making Huge Pages
Actually Useful”
(https://dl.acm.org/citation.cfm?id=3173203) helpful. The paper is
trying to minimize the number of page blocks that have both moveable and
non-moveable pages.
--
Best Regards
Yan Zi
