From: Yury Norov <yury.norov@xxxxxxxxx> Subject: mm/percpu: micro-optimize pcpu_is_populated() bitmap_next_clear_region() calls find_next_zero_bit() and find_next_bit() sequentially to find a range of clear bits. In case of pcpu_is_populated() there's a chance to return earlier if bitmap has all bits set. Link: https://lkml.kernel.org/r/20210814211713.180533-15-yury.norov@xxxxxxxxx Signed-off-by: Yury Norov <yury.norov@xxxxxxxxx> Tested-by: Wolfram Sang <wsa+renesas@xxxxxxxxxxxxxxxxxxxx> Acked-by: Dennis Zhou <dennis@xxxxxxxxxx> Cc: Alexander Lobakin <alobakin@xxxxx> Cc: Alexey Klimov <aklimov@xxxxxxxxxx> Cc: Andy Shevchenko <andriy.shevchenko@xxxxxxxxxxxxxxx> Cc: Jiri Olsa <jolsa@xxxxxxxxxx> Cc: Ulf Hansson <ulf.hansson@xxxxxxxxxx> Cc: Will Deacon <will@xxxxxxxxxx> Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> --- mm/percpu.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) --- a/mm/percpu.c~mm-percpu-micro-optimize-pcpu_is_populated +++ a/mm/percpu.c @@ -1070,17 +1070,18 @@ static void pcpu_block_update_hint_free( static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits, int *next_off) { - unsigned int page_start, page_end, rs, re; + unsigned int start, end; - page_start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE); - page_end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE); + start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE); + end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE); - rs = page_start; - bitmap_next_clear_region(chunk->populated, &rs, &re, page_end); - if (rs >= page_end) + start = find_next_zero_bit(chunk->populated, end, start); + if (start >= end) return true; - *next_off = re * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE; + end = find_next_bit(chunk->populated, end, start + 1); + + *next_off = end * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE; return false; } _
