Note that the PTE is
always writable, and we can always create a writable-device-exclusive
entry.
With this change, device-exclusive is fully compatible with THPs /
large folios. We still require PMD-sized THPs to get PTE-mapped, and
supporting PMD-mapped THP (without the PTE-remapping) is a different
endeavour that might not be worth it at this point.
I'm not sure we actually want hugepages for device exclusive, since it has
an impact on what's allowed and what not. If we only ever do 4k entries
then userspace can assume that as long atomics are separated by a 4k page
there's no issue when both the gpu and cpu hammer on them. If we try to
keep thp entries then suddenly a workload that worked before will result
in endless ping-pong between gpu and cpu because the separate atomic
counters (or whatever) now all sit in the same 2m page.
Agreed. And the conversion + mapping into the device gets trickier.
So going with thp might result in userspace having to spread out atomics
even more, which is just wasting memory and not saving any tlb entries
since often you don't need that many.
tldr; I think not supporting thp entries for device exclusive is a
feature, not a bug.
So, you agree with my "different endeavour that might not be worth it"
statement?
Yes.
Well I think we should go further and clearly document that we
intentionally return split pages. Because it's part of the uapi contract
with users of all this.
Yes, see my reply to patch #3/
And if someone needs pmd entries for performance or whatever, we need two
things:
a) userspace must mmap that memory as hugepage memory, to clearly signal
the promise that atomics are split up on hugepage sizes and not just page
size
b) we need to extend make_device_exclusive and drivers to handle the
hugetlb folio case
I think thp is simply not going to work here, it's impossible (without
potentially causing fault storms) to figure out what userspace might want.
Right, I added a link to this discussion in the patch.
Thanks!
--
Cheers,
David / dhildenb
