Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx> writes:
> On Wed, Sep 20, 2023 at 02:18:47PM +0800, Huang Ying wrote:
>> In commit f26b3fa04611 ("mm/page_alloc: limit number of high-order
>> pages on PCP during bulk free"), the PCP (Per-CPU Pageset) will be
>> drained when PCP is mostly used for high-order pages freeing to
>> improve the cache-hot pages reusing between page allocation and
>> freeing CPUs.
>>
>> But, the PCP draining mechanism may be triggered unexpectedly when
>> process exits. With some customized trace point, it was found that
>> PCP draining (free_high == true) was triggered with the order-1 page
>> freeing with the following call stack,
>>
>> => free_unref_page_commit
>> => free_unref_page
>> => __mmdrop
>> => exit_mm
>> => do_exit
>> => do_group_exit
>> => __x64_sys_exit_group
>> => do_syscall_64
>>
>> Checking the source code, this is the page table PGD
>> freeing (mm_free_pgd()). It's a order-1 page freeing if
>> CONFIG_PAGE_TABLE_ISOLATION=y. Which is a common configuration for
>> security.
>>
>> Just before that, page freeing with the following call stack was
>> found,
>>
>> => free_unref_page_commit
>> => free_unref_page_list
>> => release_pages
>> => tlb_batch_pages_flush
>> => tlb_finish_mmu
>> => exit_mmap
>> => __mmput
>> => exit_mm
>> => do_exit
>> => do_group_exit
>> => __x64_sys_exit_group
>> => do_syscall_64
>>
>> So, when a process exits,
>>
>> - a large number of user pages of the process will be freed without
>> page allocation, it's highly possible that pcp->free_factor becomes
>> > 0.
>>
>> - after freeing all user pages, the PGD will be freed, which is a
>> order-1 page freeing, PCP will be drained.
>>
>> All in all, when a process exits, it's high possible that the PCP will
>> be drained. This is an unexpected behavior.
>>
>> To avoid this, in the patch, the PCP draining will only be triggered
>> for 2 consecutive high-order page freeing.
>>
>> On a 2-socket Intel server with 224 logical CPU, we tested kbuild on
>> one socket with `make -j 112`. With the patch, the build time
>> decreases 3.4% (from 206s to 199s). The cycles% of the spinlock
>> contention (mostly for zone lock) decreases from 43.6% to 40.3% (with
>> PCP size == 361). The number of PCP draining for high order pages
>> freeing (free_high) decreases 50.8%.
>>
>> This helps network workload too for reduced zone lock contention. On
>> a 2-socket Intel server with 128 logical CPU, with the patch, the
>> network bandwidth of the UNIX (AF_UNIX) test case of lmbench test
>> suite with 16-pair processes increase 17.1%. The cycles% of the
>> spinlock contention (mostly for zone lock) decreases from 50.0% to
>> 45.8%. The number of PCP draining for high order pages
>> freeing (free_high) decreases 27.4%. The cache miss rate keeps 0.3%.
>>
>> Signed-off-by: "Huang, Ying" <ying.huang@xxxxxxxxx>
>
> Acked-by: Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx>
>
> However, I want to note that batching on exit is not necessarily
> unexpected. For processes that are multi-TB in size, the time to exit
> can actually be quite large and batching is of benefit but optimising
> for exit is rarely a winning strategy. The pattern of "all allocs on CPU
> B and all frees on CPU B" or "short-lived tasks triggering a premature
> drain" is a bit more compelling but not worth a changelog rewrite.
>>
>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>> index 4106fbc5b4b3..64d5ed2bb724 100644
>> --- a/include/linux/mmzone.h
>> +++ b/include/linux/mmzone.h
>> @@ -676,12 +676,15 @@ enum zone_watermarks {
>> #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost)
>> #define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost)
>>
>> +#define PCPF_PREV_FREE_HIGH_ORDER 0x01
>> +
>
> The meaning of the flag and its intent should have been documented.
Sure. Will add comments for the flags.
--
Best Regards,
Huang, Ying
