On 12/28/2022 9:24 AM, David Rientjes wrote: > On Fri, 23 Dec 2022, Yin Fengwei wrote: > >> Kernel build regression with LLVM was reported here: >> https://lore.kernel.org/all/Y1GCYXGtEVZbcv%2F5@dev-arch.thelio-3990X/ >> with commit f35b5d7d676e ("mm: align larger anonymous mappings on THP >> boundaries"). And the commit f35b5d7d676e was reverted. >> >> It turned out the regression is related with madvise(MADV_DONTNEED) >> was used by ld.lld. But with none PMD_SIZE aligned parameter len. >> trace-bpfcc captured: >> 531607 531732 ld.lld do_madvise.part.0 start: 0x7feca9000000, len: 0x7fb000, behavior: 0x4 >> 531607 531793 ld.lld do_madvise.part.0 start: 0x7fec86a00000, len: 0x7fb000, behavior: 0x4 >> >> If the underneath physical page is THP, the madvise(MADV_DONTNNED) can >> trigger split_queue_lock contention raised significantly. perf showed >> following data: >> 14.85% 0.00% ld.lld [kernel.kallsyms] [k] >> entry_SYSCALL_64_after_hwframe >> 11.52% >> entry_SYSCALL_64_after_hwframe >> do_syscall_64 >> __x64_sys_madvise >> do_madvise.part.0 >> zap_page_range >> unmap_single_vma >> unmap_page_range >> page_remove_rmap >> deferred_split_huge_page >> __lock_text_start >> native_queued_spin_lock_slowpath >> >> If THP can't be removed from rmap as whole THP, partial THP will be >> removed from rmap by removing sub-pages from rmap. Even the THP >> head page is added to deferred queue already, the split_queue_lock >> will be acquired and check whether the THP head page is in the queue >> already. Thus, the contention of split_queue_lock is raised. >> >> Before acquire split_queue_lock, check and bail out early if the THP >> head page is in the queue already. The checking without holding >> split_queue_lock could race with deferred_split_scan, but it doesn't >> impact the correctness here. >> >> Test result of building kernel with ld.lld: >> commit 7b5a0b664ebe (parent commit of f35b5d7d676e): >> time -f "\t%E real,\t%U user,\t%S sys" make LD=ld.lld -skj96 allmodconfig all >> 6:07.99 real, 26367.77 user, 5063.35 sys >> >> commit f35b5d7d676e: >> time -f "\t%E real,\t%U user,\t%S sys" make LD=ld.lld -skj96 allmodconfig all >> 7:22.15 real, 26235.03 user, 12504.55 sys >> >> commit f35b5d7d676e with the fixing patch: >> time -f "\t%E real,\t%U user,\t%S sys" make LD=ld.lld -skj96 allmodconfig all >> 6:08.49 real, 26520.15 user, 5047.91 sys >> >> Signed-off-by: Yin Fengwei <fengwei.yin@xxxxxxxxx> > > Acked-by: David Rientjes <rientjes@xxxxxxxxxx> Thanks for reviewing the patch. Regards Yin, Fengwei
