On 11/10/21 8:56 PM, Jason Gunthorpe wrote:
On Wed, Nov 10, 2021 at 06:54:13PM +0800, Qi Zheng wrote:
In this patch series, we add a pte_refcount field to the struct page of page
table to track how many users of PTE page table. Similar to the mechanism of
page refcount, the user of PTE page table should hold a refcount to it before
accessing. The PTE page table page will be freed when the last refcount is
dropped.
So, this approach basically adds two atomics on every PTE map
If I have it right the reason that zap cannot clean the PTEs today is
because zap cannot obtain the mmap lock due to a lock ordering issue
with the inode lock vs mmap lock.
Currently, both MADV_DONTNEED and MADV_FREE obtain the read side of
mmap_lock instead of write side, which is the reason that
jemalloc/tcmalloc prefer to use madvise() to release physical memory.
If it could obtain the mmap lock then it could do the zap using the
write side as unmapping a vma does.
Even if it obtains the write side of mmap_lock, how to make sure that
all the page table entries are empty? Traverse 512 entries every time?
Rather than adding a new "lock" to ever PTE I wonder if it would be
more efficient to break up the mmap lock and introduce a specific
rwsem for the page table itself, in addition to the PTL. Currently the
mmap lock is protecting both the vma list and the page table.
Now each level of page table has its own spin lock. Can you explain the
working mechanism of this special rwsem more clearly?
If we can reduce the protection range of mmap_lock, it is indeed a great
thing, but I think it is very difficult, and it will not solve the
problem of how to check that all entries in the page table page are
empty.
I think that would allow the lock ordering issue to be resolved and
zap could obtain a page table rwsem.
Compared to two atomics per PTE this would just be two atomic per
page table walk operation, it is conceptually a lot simpler, and would
allow freeing all the page table levels, not just PTEs.
The reason why only the PTE page is released now is that it is the
largest. This reference count can actually be used for other levels of
page tables.
?
Jason
