From: Alexander Duyck <alexander.h.duyck@xxxxxxxxxxxxxxx>
Since we are holding a reference to the page much sooner in
isolate_migratepages_block we could move the PageCompound check out of the
LRU locked section and instead just place it after get_page_unless_zero. By
doing this we can allow any of the items that might trigger a failure to
trigger a failure for the compound page rather than the order 0 page and as
a result we should be able to process the pageblock faster.
In addition by testing for PageCompound sooner we can avoid having the LRU
flag cleared and then reset in the exception case. As a result this should
prevent any possible races where another thread might be attempting to pull
the LRU pages from the list.
Signed-off-by: Alexander Duyck <alexander.h.duyck@xxxxxxxxxxxxxxx>
---
mm/compaction.c | 33 ++++++++++++++++++---------------
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index c1e9918f9dd4..3803f129fd6a 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -954,6 +954,24 @@ static bool too_many_isolated(pg_data_t *pgdat)
if (unlikely(!get_page_unless_zero(page)))
goto isolate_fail;
+ /*
+ * Page is compound. We know the order before we know if it is
+ * on the LRU so we cannot assume it is THP. However since the
+ * page will have the LRU validated shortly we can use the value
+ * to skip over this page for now or validate the LRU is set and
+ * then isolate the entire compound page if we are isolating to
+ * generate a CMA page.
+ */
+ if (PageCompound(page)) {
+ const unsigned int order = compound_order(page);
+
+ if (likely(order < MAX_ORDER))
+ low_pfn += (1UL << order) - 1;
+
+ if (!cc->alloc_contig)
+ goto isolate_fail_put;
+ }
+
if (__isolate_lru_page_prepare(page, isolate_mode) != 0)
goto isolate_fail_put;
@@ -978,23 +996,8 @@ static bool too_many_isolated(pg_data_t *pgdat)
rcu_read_unlock();
lruvec_memcg_debug(lruvec, page);
-
- /*
- * Page become compound since the non-locked check,
- * and it's on LRU. It can only be a THP so the order
- * is safe to read and it's 0 for tail pages.
- */
- if (unlikely(PageCompound(page) && !cc->alloc_contig)) {
- low_pfn += compound_nr(page) - 1;
- SetPageLRU(page);
- goto isolate_fail_put;
- }
}
- /* The whole page is taken off the LRU; skip the tail pages. */
- if (PageCompound(page))
- low_pfn += compound_nr(page) - 1;
-
/* Successfully isolated */
del_page_from_lru_list(page, lruvec, page_lru(page));
mod_node_page_state(page_pgdat(page),
