Hi, Laurent
2018-03-14 1:59 GMT+08:00 Laurent Dufour <ldufour@xxxxxxxxxxxxxxxxxx>:
> This is a port on kernel 4.16 of the work done by Peter Zijlstra to
> handle page fault without holding the mm semaphore [1].
>
> The idea is to try to handle user space page faults without holding the
> mmap_sem. This should allow better concurrency for massively threaded
> process since the page fault handler will not wait for other threads memory
> layout change to be done, assuming that this change is done in another part
> of the process's memory space. This type page fault is named speculative
> page fault. If the speculative page fault fails because of a concurrency is
> detected or because underlying PMD or PTE tables are not yet allocating, it
> is failing its processing and a classic page fault is then tried.
>
> The speculative page fault (SPF) has to look for the VMA matching the fault
> address without holding the mmap_sem, this is done by introducing a rwlock
> which protects the access to the mm_rb tree. Previously this was done using
> SRCU but it was introducing a lot of scheduling to process the VMA's
> freeing
> operation which was hitting the performance by 20% as reported by Kemi Wang
> [2].Using a rwlock to protect access to the mm_rb tree is limiting the
> locking contention to these operations which are expected to be in a O(log
> n)
> order. In addition to ensure that the VMA is not freed in our back a
> reference count is added and 2 services (get_vma() and put_vma()) are
> introduced to handle the reference count. When a VMA is fetch from the RB
> tree using get_vma() is must be later freeed using put_vma(). Furthermore,
> to allow the VMA to be used again by the classic page fault handler a
> service is introduced can_reuse_spf_vma(). This service is expected to be
> called with the mmap_sem hold. It checked that the VMA is still matching
> the specified address and is releasing its reference count as the mmap_sem
> is hold it is ensure that it will not be freed in our back. In general, the
> VMA's reference count could be decremented when holding the mmap_sem but it
> should not be increased as holding the mmap_sem is ensuring that the VMA is
> stable. I can't see anymore the overhead I got while will-it-scale
> benchmark anymore.
>
> The VMA's attributes checked during the speculative page fault processing
> have to be protected against parallel changes. This is done by using a per
> VMA sequence lock. This sequence lock allows the speculative page fault
> handler to fast check for parallel changes in progress and to abort the
> speculative page fault in that case.
>
> Once the VMA is found, the speculative page fault handler would check for
> the VMA's attributes to verify that the page fault has to be handled
> correctly or not. Thus the VMA is protected through a sequence lock which
> allows fast detection of concurrent VMA changes. If such a change is
> detected, the speculative page fault is aborted and a *classic* page fault
> is tried. VMA sequence lockings are added when VMA attributes which are
> checked during the page fault are modified.
>
> When the PTE is fetched, the VMA is checked to see if it has been changed,
> so once the page table is locked, the VMA is valid, so any other changes
> leading to touching this PTE will need to lock the page table, so no
> parallel change is possible at this time.
>
> The locking of the PTE is done with interrupts disabled, this allows to
> check for the PMD to ensure that there is not an ongoing collapsing
> operation. Since khugepaged is firstly set the PMD to pmd_none and then is
> waiting for the other CPU to have catch the IPI interrupt, if the pmd is
> valid at the time the PTE is locked, we have the guarantee that the
> collapsing opertion will have to wait on the PTE lock to move foward. This
> allows the SPF handler to map the PTE safely. If the PMD value is different
> than the one recorded at the beginning of the SPF operation, the classic
> page fault handler will be called to handle the operation while holding the
> mmap_sem. As the PTE lock is done with the interrupts disabled, the lock is
> done using spin_trylock() to avoid dead lock when handling a page fault
> while a TLB invalidate is requested by an other CPU holding the PTE.
>
> Support for THP is not done because when checking for the PMD, we can be
> confused by an in progress collapsing operation done by khugepaged. The
> issue is that pmd_none() could be true either if the PMD is not already
> populated or if the underlying PTE are in the way to be collapsed. So we
> cannot safely allocate a PMD if pmd_none() is true.
>
> This series a new software performance event named 'speculative-faults' or
> 'spf'. It counts the number of successful page fault event handled in a
> speculative way. When recording 'faults,spf' events, the faults one is
> counting the total number of page fault events while 'spf' is only counting
> the part of the faults processed in a speculative way.
>
> There are some trace events introduced by this series. They allow to
> identify why the page faults where not processed in a speculative way. This
> doesn't take in account the faults generated by a monothreaded process
> which directly processed while holding the mmap_sem. This trace events are
> grouped in a system named 'pagefault', they are:
> - pagefault:spf_pte_lock : if the pte was already locked by another thread
> - pagefault:spf_vma_changed : if the VMA has been changed in our back
> - pagefault:spf_vma_noanon : the vma->anon_vma field was not yet set.
> - pagefault:spf_vma_notsup : the VMA's type is not supported
> - pagefault:spf_vma_access : the VMA's access right are not respected
> - pagefault:spf_pmd_changed : the upper PMD pointer has changed in our
> back.
>
> To record all the related events, the easier is to run perf with the
> following arguments :
> $ perf stat -e 'faults,spf,pagefault:*' <command>
>
> This series builds on top of v4.16-rc2-mmotm-2018-02-21-14-48 and is
> functional on x86 and PowerPC.
>
> ---------------------
> Real Workload results
>
> As mentioned in previous email, we did non official runs using a "popular
> in memory multithreaded database product" on 176 cores SMT8 Power system
> which showed a 30% improvements in the number of transaction processed per
> second. This run has been done on the v6 series, but changes introduced in
> this new verion should not impact the performance boost seen.
>
> Here are the perf data captured during 2 of these runs on top of the v8
> series:
> vanilla spf
> faults 89.418 101.364
> spf n/a 97.989
>
> With the SPF kernel, most of the page fault were processed in a speculative
> way.
>
> ------------------
> Benchmarks results
>
> Base kernel is v4.16-rc4-mmotm-2018-03-09-16-34
> SPF is BASE + this series
>
> Kernbench:
> ----------
> Here are the results on a 16 CPUs X86 guest using kernbench on a 4.13-rc4
> kernel (kernel is build 5 times):
>
> Average Half load -j 8
> Run (std deviation)
> BASE SPF
> Elapsed Time 151.36 (1.40139) 151.748 (1.09716) 0.26%
> User Time 1023.19 (3.58972) 1027.35 (2.30396) 0.41%
> System Time 125.026 (1.8547) 124.504 (0.980015) -0.42%
> Percent CPU 758.2 (5.54076) 758.6 (3.97492) 0.05%
> Context Switches 54924 (453.634) 54851 (382.293) -0.13%
> Sleeps 105589 (704.581) 105282 (435.502) -0.29%
>
> Average Optimal load -j 16
> Run (std deviation)
> BASE SPF
> Elapsed Time 74.804 (1.25139) 74.368 (0.406288) -0.58%
> User Time 962.033 (64.5125) 963.93 (66.8797) 0.20%
> System Time 110.771 (15.0817) 110.387 (14.8989) -0.35%
> Percent CPU 1045.7 (303.387) 1049.1 (306.255) 0.33%
> Context Switches 76201.8 (22433.1) 76170.4 (22482.9) -0.04%
> Sleeps 110289 (5024.05) 110220 (5248.58) -0.06%
>
> During a run on the SPF, perf events were captured:
> Performance counter stats for '../kernbench -M':
> 510334017 faults
> 200 spf
> 0 pagefault:spf_pte_lock
> 0 pagefault:spf_vma_changed
> 0 pagefault:spf_vma_noanon
> 2174 pagefault:spf_vma_notsup
> 0 pagefault:spf_vma_access
> 0 pagefault:spf_pmd_changed
>
> Very few speculative page fault were recorded as most of the processes
> involved are monothreaded (sounds that on this architecture some threads
> were created during the kernel build processing).
>
> Here are the kerbench results on a 80 CPUs Power8 system:
>
> Average Half load -j 40
> Run (std deviation)
> BASE SPF
> Elapsed Time 116.958 (0.73401) 117.43 (0.927497) 0.40%
> User Time 4472.35 (7.85792) 4480.16 (19.4909) 0.17%
> System Time 136.248 (0.587639) 136.922 (1.09058) 0.49%
> Percent CPU 3939.8 (20.6567) 3931.2 (17.2829) -0.22%
> Context Switches 92445.8 (236.672) 92720.8 (270.118) 0.30%
> Sleeps 318475 (1412.6) 317996 (1819.07) -0.15%
>
> Average Optimal load -j 80
> Run (std deviation)
> BASE SPF
> Elapsed Time 106.976 (0.406731) 107.72 (0.329014) 0.70%
> User Time 5863.47 (1466.45) 5865.38 (1460.27) 0.03%
> System Time 159.995 (25.0393) 160.329 (24.6921) 0.21%
> Percent CPU 5446.2 (1588.23) 5416 (1565.34) -0.55%
> Context Switches 223018 (137637) 224867 (139305) 0.83%
> Sleeps 330846 (13127.3) 332348 (15556.9) 0.45%
>
> During a run on the SPF, perf events were captured:
> Performance counter stats for '../kernbench -M':
> 116612488 faults
> 0 spf
> 0 pagefault:spf_pte_lock
> 0 pagefault:spf_vma_changed
> 0 pagefault:spf_vma_noanon
> 473 pagefault:spf_vma_notsup
> 0 pagefault:spf_vma_access
> 0 pagefault:spf_pmd_changed
>
> Most of the processes involved are monothreaded so SPF is not activated but
> there is no impact on the performance.
>
> Ebizzy:
> -------
> The test is counting the number of records per second it can manage, the
> higher is the best. I run it like this 'ebizzy -mTRp'. To get consistent
> result I repeated the test 100 times and measure the average result. The
> number is the record processes per second, the higher is the best.
>
> BASE SPF delta
> 16 CPUs x86 VM 14902.6 95905.16 543.55%
> 80 CPUs P8 node 37240.24 78185.67 109.95%
>
> Here are the performance counter read during a run on a 16 CPUs x86 VM:
> Performance counter stats for './ebizzy -mRTp':
> 888157 faults
> 884773 spf
> 92 pagefault:spf_pte_lock
> 2379 pagefault:spf_vma_changed
> 0 pagefault:spf_vma_noanon
> 80 pagefault:spf_vma_notsup
> 0 pagefault:spf_vma_access
> 0 pagefault:spf_pmd_changed
>
> And the ones captured during a run on a 80 CPUs Power node:
> Performance counter stats for './ebizzy -mRTp':
> 762134 faults
> 728663 spf
> 19101 pagefault:spf_pte_lock
> 13969 pagefault:spf_vma_changed
> 0 pagefault:spf_vma_noanon
> 272 pagefault:spf_vma_notsup
> 0 pagefault:spf_vma_access
> 0 pagefault:spf_pmd_changed
>
> In ebizzy's case most of the page fault were handled in a speculative way,
> leading the ebizzy performance boost.
We ported the SPF to kernel 4.9 in android devices.
For the app launch time, It improves about 15% average. For the apps
which have hundreds of threads, it will be about 20%.
Thanks.
>
> ------------------
> Changes since v8:
> - Don't check PMD when locking the pte when THP is disabled
> Thanks to Daniel Jordan for reporting this.
> - Rebase on 4.16
> Changes since v7:
> - move pte_map_lock() and pte_spinlock() upper in mm/memory.c (patch 4 &
> 5)
> - make pte_unmap_same() compatible with the speculative page fault (patch
> 6)
> Changes since v6:
> - Rename config variable to CONFIG_SPECULATIVE_PAGE_FAULT (patch 1)
> - Review the way the config variable is set (patch 1 to 3)
> - Introduce mm_rb_write_*lock() in mm/mmap.c (patch 18)
> - Merge patch introducing pte try locking in the patch 18.
> Changes since v5:
> - use rwlock agains the mm RB tree in place of SRCU
> - add a VMA's reference count to protect VMA while using it without
> holding the mmap_sem.
> - check PMD value to detect collapsing operation
> - don't try speculative page fault for mono threaded processes
> - try to reuse the fetched VMA if VM_RETRY is returned
> - go directly to the error path if an error is detected during the SPF
> path
> - fix race window when moving VMA in move_vma()
> Changes since v4:
> - As requested by Andrew Morton, use CONFIG_SPF and define it earlier in
> the series to ease bisection.
> Changes since v3:
> - Don't build when CONFIG_SMP is not set
> - Fixed a lock dependency warning in __vma_adjust()
> - Use READ_ONCE to access p*d values in handle_speculative_fault()
> - Call memcp_oom() service in handle_speculative_fault()
> Changes since v2:
> - Perf event is renamed in PERF_COUNT_SW_SPF
> - On Power handle do_page_fault()'s cleaning
> - On Power if the VM_FAULT_ERROR is returned by
> handle_speculative_fault(), do not retry but jump to the error path
> - If VMA's flags are not matching the fault, directly returns
> VM_FAULT_SIGSEGV and not VM_FAULT_RETRY
> - Check for pud_trans_huge() to avoid speculative path
> - Handles _vm_normal_page()'s introduced by 6f16211df3bf
> ("mm/device-public-memory: device memory cache coherent with CPU")
> - add and review few comments in the code
> Changes since v1:
> - Remove PERF_COUNT_SW_SPF_FAILED perf event.
> - Add tracing events to details speculative page fault failures.
> - Cache VMA fields values which are used once the PTE is unlocked at the
> end of the page fault events.
> - Ensure that fields read during the speculative path are written and read
> using WRITE_ONCE and READ_ONCE.
> - Add checks at the beginning of the speculative path to abort it if the
> VMA is known to not be supported.
> Changes since RFC V5 [5]
> - Port to 4.13 kernel
> - Merging patch fixing lock dependency into the original patch
> - Replace the 2 parameters of vma_has_changed() with the vmf pointer
> - In patch 7, don't call __do_fault() in the speculative path as it may
> want to unlock the mmap_sem.
> - In patch 11-12, don't check for vma boundaries when
> page_add_new_anon_rmap() is called during the spf path and protect against
> anon_vma pointer's update.
> - In patch 13-16, add performance events to report number of successful
> and failed speculative events.
>
> [1]
> http://linux-kernel.2935.n7.nabble.com/RFC-PATCH-0-6-Another-go-at-speculative-page-faults-tt965642.html#none
> [2] https://patchwork.kernel.org/patch/9999687/
>
>
> Laurent Dufour (20):
> mm: Introduce CONFIG_SPECULATIVE_PAGE_FAULT
> x86/mm: Define CONFIG_SPECULATIVE_PAGE_FAULT
> powerpc/mm: Define CONFIG_SPECULATIVE_PAGE_FAULT
> mm: Introduce pte_spinlock for FAULT_FLAG_SPECULATIVE
> mm: make pte_unmap_same compatible with SPF
> mm: Protect VMA modifications using VMA sequence count
> mm: protect mremap() against SPF hanlder
> mm: Protect SPF handler against anon_vma changes
> mm: Cache some VMA fields in the vm_fault structure
> mm/migrate: Pass vm_fault pointer to migrate_misplaced_page()
> mm: Introduce __lru_cache_add_active_or_unevictable
> mm: Introduce __maybe_mkwrite()
> mm: Introduce __vm_normal_page()
> mm: Introduce __page_add_new_anon_rmap()
> mm: Protect mm_rb tree with a rwlock
> mm: Adding speculative page fault failure trace events
> perf: Add a speculative page fault sw event
> perf tools: Add support for the SPF perf event
> mm: Speculative page fault handler return VMA
> powerpc/mm: Add speculative page fault
>
> Peter Zijlstra (4):
> mm: Prepare for FAULT_FLAG_SPECULATIVE
> mm: VMA sequence count
> mm: Provide speculative fault infrastructure
> x86/mm: Add speculative pagefault handling
>
> arch/powerpc/Kconfig | 1 +
> arch/powerpc/mm/fault.c | 31 +-
> arch/x86/Kconfig | 1 +
> arch/x86/mm/fault.c | 38 ++-
> fs/proc/task_mmu.c | 5 +-
> fs/userfaultfd.c | 17 +-
> include/linux/hugetlb_inline.h | 2 +-
> include/linux/migrate.h | 4 +-
> include/linux/mm.h | 92 +++++-
> include/linux/mm_types.h | 7 +
> include/linux/pagemap.h | 4 +-
> include/linux/rmap.h | 12 +-
> include/linux/swap.h | 10 +-
> include/trace/events/pagefault.h | 87 +++++
> include/uapi/linux/perf_event.h | 1 +
> kernel/fork.c | 3 +
> mm/Kconfig | 3 +
> mm/hugetlb.c | 2 +
> mm/init-mm.c | 3 +
> mm/internal.h | 20 ++
> mm/khugepaged.c | 5 +
> mm/madvise.c | 6 +-
> mm/memory.c | 594 ++++++++++++++++++++++++++++++----
> mm/mempolicy.c | 51 ++-
> mm/migrate.c | 4 +-
> mm/mlock.c | 13 +-
> mm/mmap.c | 211 +++++++++---
> mm/mprotect.c | 4 +-
> mm/mremap.c | 13 +
> mm/rmap.c | 5 +-
> mm/swap.c | 6 +-
> mm/swap_state.c | 8 +-
> tools/include/uapi/linux/perf_event.h | 1 +
> tools/perf/util/evsel.c | 1 +
> tools/perf/util/parse-events.c | 4 +
> tools/perf/util/parse-events.l | 1 +
> tools/perf/util/python.c | 1 +
> 37 files changed, 1097 insertions(+), 174 deletions(-)
> create mode 100644 include/trace/events/pagefault.h
>
> --
> 2.7.4
>
