Now CCing the correct Willy :)
On 11.12.24 12:55, David Hildenbrand wrote:
Hi,
PageAnonExclusive (PAE) is working very reliable at this point. But
especially in the context of THPs (large folios) we'd like to do better:
(1) For PTE-mapped THP, we have to maintain it per page. We'd like to
avoid per-page flags as good as possible (e.g., waste in "struct
page", touching many cachelines).
(2) We currently have to use atomics to set/clear the flag, even when
working on tail pages. While there would be ways to mitigate this
when modifying the flags of multiple tail pages (bitlock protecting
all tail page flag updates), I'd much rather avoid messing with
page tail flags at all.
In general, the PAE bit can be considered an extended PTE bit that we
currently store in the "struct page" that is mapped by the PTE. Ideally,
we'd just store that information in the PTE, or alongside the PTE:
A writable PTE implies PAE. A write-protected PTE needs additional
information whether it is PAE (-> whether we can just remap it writable,
FOLL_FORCE to it, PIN it ...).
We are out of PTE bits, especially when having to implement it across
*all* architectures. That's one of the reasons we went with PAE back
then. As a nice side-effect it allowed for sanity checks when unpinning
folios (-> PAE must still be set, which is impossible when the
information stored in the PTE).
There are 3 main approaches I've been looking into:
(A) Make it a per-folio flag. I've spent endless hours trying to get it
conceptually right, but it's just a big pain: as soon as we clear
the flag, we have to make sure that all PTEs are write-protected,
that the folio is not pinned, and that concurrent GUP cannot work.
So far the page table lock protected the PAE bit, but with a per-
folio flag that is not guaranteed for THPs.
fork() with things like VM_DONTCOPY, VM_DONTFORK, early-abort, page
migration/swapout that can happen any time during fork etc. make
this really though to get right with THPs. My head hurts any time I
think about it.
While I think fork() itself can be handled, the concurrent page
migration / swapout is where it gets extremely tricky.
This can be done somehow I'm sure, but the devil is in the corner
cases when having multiple PTEs mapping a large folio. We'd still
need atomics to set/clear the single folio flag, because of the
nature of concurrent folio flag updates.
(B) Allocate additional metadata (PAE bitmap) for page tables, protected
by the PTL. That is, when we want to *clear* PAE (fork(), KSM), we'd
lazily allocate the bitmap and store it for our page table.
On x86: 512 PTEs -> 512bits -> 64byte
fork() gets a bit more expensive, because we'd have to allocate this
bitmap for the parent and the child page table we are working on, so
we can mark the pages as "!PAE" in both page tables.
This could work, I have not prototyped it. We'd have to support it
on the PTE/PMD/PUD-table level.
One tricky thing is having multiple pagetables per page, but I
assume it can be handled (we should have a single PT lock for all of
them IIRC, and only need to address the bitmap at the right offset).
Another challenge is how to link to this metadata from ptdesc on all
archs.. So far, __page_mapping is unused, and could maybe be used to
link to such metadata -- at least page tables can be identified
reliably using the page type.
(C) Encode it in the PTE.
pte_write() -> PAE
!pte_write() && pte_dirty() -> PAE
!pte_write && !pte_dirty() -> !PAE
That implies, that when wrprotecting a PTE, we'd have to move the
dirty bit to the folio. When wr-unprotecting it, we could mark the
PTE dirty if the folio is dirty.
I suspect that most anon folios are dirty most of the time either
way, and the common case of having them just writable in the PTE
wouldn't change.
The main idea is that nobody (including HW) should ever be marking a
readonly PTE dirty (so the theory behind it). We have to take good
care whenever we modify/query the dirty bit or modify the writable
bit.
There is quite some code to audit/sanitize. Further, we'd have to
decouple softdirty PTE handling from dirty PTE handling (pte_mkdirty
sets the pte softdirty), and adjust arm64 cont-pte and similar PTE
batching code to respect the per-PTE dirty bit when
the PTE is write-protected.
This would be the most elegant solution, but requires a bit of care
+ sanity checks.
Any thoughts or other ideas?
--
Cheers,
David / dhildenb
