Le 17/01/2019 à 16:51, zhong jiang a écrit :
On 2019/1/16 19:41, Vinayak Menon wrote:
On 1/15/2019 1:54 PM, Laurent Dufour wrote:
Le 14/01/2019 à 14:19, Vinayak Menon a écrit :
On 1/11/2019 9:13 PM, Vinayak Menon wrote:
Hi Laurent,
We are observing an issue with speculative page fault with the following test code on ARM64 (4.14 kernel, 8 cores).
With the patch below, we don't hit the issue.
From: Vinayak Menon <vinmenon@xxxxxxxxxxxxxx>
Date: Mon, 14 Jan 2019 16:06:34 +0530
Subject: [PATCH] mm: flush stale tlb entries on speculative write fault
It is observed that the following scenario results in
threads A and B of process 1 blocking on pthread_mutex_lock
forever after few iterations.
CPU 1 CPU 2 CPU 3
Process 1, Process 1, Process 1,
Thread A Thread B Thread C
while (1) { while (1) { while(1) {
pthread_mutex_lock(l) pthread_mutex_lock(l) fork
pthread_mutex_unlock(l) pthread_mutex_unlock(l) }
} }
When from thread C, copy_one_pte write-protects the parent pte
(of lock l), stale tlb entries can exist with write permissions
on one of the CPUs at least. This can create a problem if one
of the threads A or B hits the write fault. Though dup_mmap calls
flush_tlb_mm after copy_page_range, since speculative page fault
does not take mmap_sem it can proceed further fixing a fault soon
after CPU 3 does ptep_set_wrprotect. But the CPU with stale tlb
entry can still modify old_page even after it is copied to
new_page by wp_page_copy, thus causing a corruption.
Nice catch and thanks for your investigation!
There is a real synchronization issue here between copy_page_range() and the speculative page fault handler. I didn't get it on PowerVM since the TLB are flushed when arch_exit_lazy_mode() is called in copy_page_range() but now, I can get it when running on x86_64.
Signed-off-by: Vinayak Menon <vinmenon@xxxxxxxxxxxxxx>
---
mm/memory.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/mm/memory.c b/mm/memory.c
index 52080e4..1ea168ff 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4507,6 +4507,13 @@ int __handle_speculative_fault(struct mm_struct *mm, unsigned long address,
return VM_FAULT_RETRY;
}
+ /*
+ * Discard tlb entries created before ptep_set_wrprotect
+ * in copy_one_pte
+ */
+ if (flags & FAULT_FLAG_WRITE && !pte_write(vmf.orig_pte))
+ flush_tlb_page(vmf.vma, address);
+
mem_cgroup_oom_enable();
ret = handle_pte_fault(&vmf);
mem_cgroup_oom_disable();
Your patch is fixing the race but I'm wondering about the cost of these tlb flushes. Here we are flushing on a per page basis (architecture like x86_64 are smarter and flush more pages) but there is a request to flush a range of tlb entries each time a cow page is newly touched. I think there could be some bad impact here.
Another option would be to flush the range in copy_pte_range() before unlocking the page table lock. This will flush entries flush_tlb_mm() would later handle in dup_mmap() but that will be called once per fork per cow VMA.
But wouldn't this cause an unnecessary impact if most of the COW pages remain untouched (which I assume would be the usual case) and thus do not create a fault ?
I tried the attached patch which seems to fix the issue on x86_64. Could you please give it a try on arm64 ?
Your patch works fine on arm64 with a minor change. Thanks Laurent.
Hi, Vinayak and Laurent
I think the below change will impact the performance significantly. Becuase most of process has many
vmas with cow flags. Flush the tlb in advance is not the better way to avoid the issue and it will
call the flush_tlb_mm later.
I think we can try the following way to do.
vm_write_begin(vma)
copy_pte_range
vm_write_end(vma)
The speculative page fault will return to grap the mmap_sem to run the nromal path.
Any thought?
Hi Zhong,
I agree that flushing the TLB could have a bad impact on the
performance, but tagging the VMA when copy_pte_range() is not fixing the
issue as the VMA must be flagged until the PTE are flushed.
Here is what happens:
CPU A CPU B CPU C
fork()
copy_pte_range()
set PTE rdonly
got to next VMA...
. PTE is seen rdonly PTE still writable
. thread is writing to page
. -> page fault
. copy the page Thread writes to page
. . -> no page fault
. update the PTE
. flush TLB for that PTE
flush TLB PTE are now rdonly
So the write done by the CPU C is interfering with the page copy
operation done by CPU B, leading to the data corruption.
Flushing the PTE in copy_pte_range() is fixing the issue as the CPU C is
seeing the PTE as rdonly earlier. But this impacts performance.
Another option, I'll work on is to flag _all the COW eligible_ VMA
before starting copying them and until the PTE are flushed on the CPU A.
This way when the CPU B will page fault the speculative handler will
abort because the VMA is in the way to be touched.
But I need to ensure that all the calls to copy_pte_range() are handling
this correctly.
Laurent.
