From: Bartlomiej Zolnierkiewicz <b.zolnierkie@xxxxxxxxxxx>
Subject: [PATCH] cma: cached pageblock type fixup
CMA pages added to per-cpu pages lists in free_hot_cold_page()
have private field set to MIGRATE_CMA pageblock type . If this
happes just before start_isolate_page_range() in alloc_contig_range()
changes pageblock type of the page to MIGRATE_ISOLATE it may result
in the cached pageblock type being stale in free_pcppages_bulk()
(which may be triggered by drain_all_pages() in alloc_contig_range()),
page being added to MIGRATE_CMA free list instead of MIGRATE_ISOLATE
one in __free_one_page() and (if the page is reused just before
test_pages_isolated() check) causing alloc_contig_range() failure.
Fix such situation by checking whether pageblock type of the page
changed to MIGRATE_ISOLATE for MIGRATE_CMA type pages in
free_pcppages_bulk() and if so fixup the pageblock type to
MIGRATE_ISOLATE (so the page will be added to MIGRATE_ISOLATE free
list in __free_one_page() and won't be used).
Similar situation can happen if rmqueue_bulk() sets cached pageblock
of the page to MIGRATE_CMA and start_isolate_page_range() is called
before buffered_rmqueue() completes (so the page may used by
get_page_from_freelist() and cause test_pages_isolated() check
failure in alloc_contig_range()). Fix it in buffered_rmqueue() by
changing the pageblock type of the affected page if needed, freeing
page back to buddy allocator and retrying the allocation.
Please note that even with this patch applied some page allocation
vs alloc_contig_range() races are still possible and may result in
rare test_pages_isolated() failures.
Cc: Michal Nazarewicz <mina86@xxxxxxxxxx>
Cc: Marek Szyprowski <m.szyprowski@xxxxxxxxxxx>
Cc: Mel Gorman <mgorman@xxxxxxx>
Signed-off-by: Bartlomiej Zolnierkiewicz <b.zolnierkie@xxxxxxxxxxx>
Signed-off-by: Kyungmin Park <kyungmin.park@xxxxxxxxxxx>
---
mm/page_alloc.c | 38 ++++++++++++++++++++++++++++++++++++--
1 file changed, 36 insertions(+), 2 deletions(-)
Index: b/mm/page_alloc.c
===================================================================
--- a/mm/page_alloc.c 2012-05-14 16:19:10.052973990 +0200
+++ b/mm/page_alloc.c 2012-05-15 12:40:54.199127705 +0200
@@ -664,12 +664,24 @@
batch_free = to_free;
do {
+ int mt;
+
page = list_entry(list->prev, struct page, lru);
/* must delete as __free_one_page list manipulates */
list_del(&page->lru);
+
+ mt = page_private(page);
+ /*
+ * cached MIGRATE_CMA pageblock type may have changed
+ * during isolation
+ */
+ if (is_migrate_cma(mt) &&
+ get_pageblock_migratetype(page) == MIGRATE_ISOLATE)
+ mt = MIGRATE_ISOLATE;
+
/* MIGRATE_MOVABLE list may include MIGRATE_RESERVEs */
- __free_one_page(page, zone, 0, page_private(page));
- trace_mm_page_pcpu_drain(page, 0, page_private(page));
+ __free_one_page(page, zone, 0, mt);
+ trace_mm_page_pcpu_drain(page, 0, mt);
} while (--to_free && --batch_free && !list_empty(list));
}
__mod_zone_page_state(zone, NR_FREE_PAGES, count);
@@ -1440,6 +1452,7 @@
if (likely(order == 0)) {
struct per_cpu_pages *pcp;
struct list_head *list;
+ int mt;
local_irq_save(flags);
pcp = &this_cpu_ptr(zone->pageset)->pcp;
@@ -1459,6 +1472,27 @@
list_del(&page->lru);
pcp->count--;
+
+ spin_lock(&zone->lock);
+ mt = page_private(page);
+ /*
+ * cached MIGRATE_CMA pageblock type may have changed
+ * during isolation
+ */
+ if ((is_migrate_cma(mt) &&
+ get_pageblock_migratetype(page) == MIGRATE_ISOLATE) ||
+ mt == MIGRATE_ISOLATE) {
+ mt = MIGRATE_ISOLATE;
+
+ zone->all_unreclaimable = 0;
+ zone->pages_scanned = 0;
+
+ __free_one_page(page, zone, 0, mt);
+ __mod_zone_page_state(zone, NR_FREE_PAGES, 1);
+ spin_unlock(&zone->lock);
+ goto again;
+ } else
+ spin_unlock(&zone->lock);
} else {
if (unlikely(gfp_flags & __GFP_NOFAIL)) {
/*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>