From: Jesper Dangaard Brouer <brouer@xxxxxxxxxx> Subject: mm/page_alloc: optimize code layout for __alloc_pages_bulk Looking at perf-report and ASM-code for __alloc_pages_bulk() it is clear that the code activated is suboptimal. The compiler guesses wrong and places unlikely code at the beginning. Due to the use of WARN_ON_ONCE() macro the UD2 asm instruction is added to the code, which confuse the I-cache prefetcher in the CPU. [mgorman@xxxxxxxxxxxxxxxxxxx: minor changes and rebasing] Link: https://lkml.kernel.org/r/20210325114228.27719-5-mgorman@xxxxxxxxxxxxxxxxxxx Signed-off-by: Jesper Dangaard Brouer <brouer@xxxxxxxxxx> Signed-off-by: Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx> Reviewed-by: Alexander Lobakin <alobakin@xxxxx> Acked-By: Vlastimil Babka <vbabka@xxxxxxx> Cc: Alexander Duyck <alexander.duyck@xxxxxxxxx> Cc: Christoph Hellwig <hch@xxxxxxxxxxxxx> Cc: Chuck Lever <chuck.lever@xxxxxxxxxx> Cc: David Miller <davem@xxxxxxxxxxxxx> Cc: Ilias Apalodimas <ilias.apalodimas@xxxxxxxxxx> Cc: Matthew Wilcox (Oracle) <willy@xxxxxxxxxxxxx> Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> --- mm/page_alloc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/mm/page_alloc.c~mm-page_alloc-optimize-code-layout-for-__alloc_pages_bulk +++ a/mm/page_alloc.c @@ -5042,7 +5042,7 @@ unsigned long __alloc_pages_bulk(gfp_t g unsigned int alloc_flags = ALLOC_WMARK_LOW; int nr_populated = 0; - if (WARN_ON_ONCE(nr_pages <= 0)) + if (unlikely(nr_pages <= 0)) return 0; /* @@ -5089,7 +5089,7 @@ unsigned long __alloc_pages_bulk(gfp_t g * If there are no allowed local zones that meets the watermarks then * try to allocate a single page and reclaim if necessary. */ - if (!zone) + if (unlikely(!zone)) goto failed; /* Attempt the batch allocation */ @@ -5107,7 +5107,7 @@ unsigned long __alloc_pages_bulk(gfp_t g page = __rmqueue_pcplist(zone, ac.migratetype, alloc_flags, pcp, pcp_list); - if (!page) { + if (unlikely(!page)) { /* Try and get at least one page */ if (!nr_populated) goto failed_irq; _
