On Fri, 31 May 2019, Michal Hocko wrote: > > The problem which this patch addresses has apparently gone unreported for > > 4+ years since > > Can we finaly stop considering the time and focus on the what is the > most reasonable behavior in general case please? Conserving mistakes > based on an argument that we have them for many years is just not > productive. It is very well possible that workloads that suffer from > this simply run on older distribution kernels which are moving towards > newer kernels very slowly. > That's fine, but we also must be mindful of users who have used MADV_HUGEPAGE over the past four years based on its hard-coded behavior that would now regress as a result. > > commit 077fcf116c8c2bd7ee9487b645aa3b50368db7e1 > > Author: Aneesh Kumar K.V <aneesh.kumar@xxxxxxxxxxxxxxxxxx> > > Date: Wed Feb 11 15:27:12 2015 -0800 > > > > mm/thp: allocate transparent hugepages on local node > > Let me quote the commit message to the full lenght > " > This make sure that we try to allocate hugepages from local node if > allowed by mempolicy. If we can't, we fallback to small page allocation > based on mempolicy. This is based on the observation that allocating > pages on local node is more beneficial than allocating hugepages on remote > node. > > With this patch applied we may find transparent huge page allocation > failures if the current node doesn't have enough freee hugepages. Before > this patch such failures result in us retrying the allocation on other > nodes in the numa node mask. > " > > I do not see any single numbers backing those claims or any mention of a > workload that would benefit from the change. Besides that, we have seen > that THP on a remote (but close) node might be performing better per > Andrea's numbers. So those claims do not apply in general. > I confirm that on every platform I have tested that the access latency to local pages of the native page size has been less than hugepages on any remote node. I think it's generally accepted that NUMA-ness is more important than huge-ness in terms of access latency and this is not the reason why the revert is being proposed. Certainly if the argument is to be made that the default behavior should be what is in the best interest of Linux users in totality, preferring remote hugepages over local pages of the native page size would not be anywhere close. I agree with Aneesh's commit message 100%. > > My goal is to reach a solution that does not cause anybody to incur > > performance penalties as a result of it. > > That is certainly appreciated and I can offer my help there as well. But > I believe we should start with a code base that cannot generate a > swapping storm by a trivial code as demonstrated by Mel. A general idea > on how to approve the situation has been already outlined for a default > case and a new memory policy has been mentioned as well but we need > something to start with and neither of the two is compatible with the > __GFP_THISNODE behavior. > Thus far, I haven't seen anybody engage in discussion on how to address the issue other than proposed reverts that readily acknowledge they cause other users to regress. If all nodes are fragmented, the swap storms that are currently reported for the local node would be made worse by the revert -- if remote hugepages cannot be faulted quickly then it's only compounded the problem. The hugepage aware mempolicy idea is one way that could describe what should be done for these allocations, we could also perhaps consider heuristics that consider the memory pressure of the local node: just as I've never seen a platform where remote hugepages have better access latency than local pages, I've never seen a platform where remote hugepages aren't a win over remote pages. This, however, is more delicate on 4 socket and 8 socket platforms but I think a general rule that a hugepage is better, if readily allocatable, than a page on the same node. (I've seen cross socket latency for hugepages match the latency for pages, so not always a win: better to leave the hugepage available remotely for something running on that node.) If the local node has a workingset that reaches its capacity, then it makes sense to fault a remote hugepage instead because otherwise we are thrashing the local node. That's not a compaction problem, though, it's a reclaim problem. If compaction fails and it's because we can't allocate target pages, it's under memory pressure and it's uncertain if reclaim will help: it may fail after expensive swap, the reclaimed pages could be grabbed by somebody else and we loop, or the compaction freeing scanner can't find it. Worst case is we thrash the local node in a swap storm. So the argument I've made when the removal of __GFP_THISNODE was first proposed is that local hugepage allocation should be the preference including direct compaction for users of MADV_HUGEPAGE (or thp enabled=always) but reclaim should never be done locally. I'd very much like to engage with anybody who would be willing to discuss fixes that work for everybody rather than only propose reverts and leave others to deal with new performance regressions.
