Hi all,
Currently huge_ptep_clear_flush on x86 is a nop. How is dtlb invalidation handled, for the copy-on-write cases on huge pages ?
I have actually verified that dtlb consistency is maintained with an example code, under the following scenario:
- parent allocates a 2M hugepage (via mmap/MAP_HUGETLB), fills page with some random data
- forks a child process
- clones a thread
(all parent, child and thread run on separate cores, pinned).
The child process as well as the thread, read the contents of the hugepage (thus there is a dtlb entry at the core that the thread runs)
At a later point, parent writes into the page, thus inducing a CoW fault. Since this is a hugepage, there is no tlb page flushing taking place (no tlb-flushing IPIs). My assumption is that the thread would be now reading from the physical page that belongs to the child after CoW, since the parent pgtable pte is now pointing to a newly allocated page, but the core executing the thread has not received any tlb invalidation interrupts (thus it would be following the "old" tlb entry). Oddly enough, this does not hold true (the thread can see the updated page).
Going through the intel devel manuals, I do not see how this would happen. It does not seem that large pages are treated differently from 4K pages as far as tlb invalidation goes.
Any ideas ? Does the kernel somehow manage this in a different way, or is it an x86 thing that is non-obvious ?
Regards,
Anthony
Currently huge_ptep_clear_flush on x86 is a nop. How is dtlb invalidation handled, for the copy-on-write cases on huge pages ?
I have actually verified that dtlb consistency is maintained with an example code, under the following scenario:
- parent allocates a 2M hugepage (via mmap/MAP_HUGETLB), fills page with some random data
- forks a child process
- clones a thread
(all parent, child and thread run on separate cores, pinned).
The child process as well as the thread, read the contents of the hugepage (thus there is a dtlb entry at the core that the thread runs)
At a later point, parent writes into the page, thus inducing a CoW fault. Since this is a hugepage, there is no tlb page flushing taking place (no tlb-flushing IPIs). My assumption is that the thread would be now reading from the physical page that belongs to the child after CoW, since the parent pgtable pte is now pointing to a newly allocated page, but the core executing the thread has not received any tlb invalidation interrupts (thus it would be following the "old" tlb entry). Oddly enough, this does not hold true (the thread can see the updated page).
Going through the intel devel manuals, I do not see how this would happen. It does not seem that large pages are treated differently from 4K pages as far as tlb invalidation goes.
Any ideas ? Does the kernel somehow manage this in a different way, or is it an x86 thing that is non-obvious ?
Regards,
Anthony