On 11.09.23 13:52, Catalin Marinas wrote:
On Wed, Sep 06, 2023 at 12:23:21PM +0100, Alexandru Elisei wrote:
On Thu, Aug 24, 2023 at 04:24:30PM +0100, Catalin Marinas wrote:
On Thu, Aug 24, 2023 at 01:25:41PM +0200, David Hildenbrand wrote:
On 24.08.23 13:06, David Hildenbrand wrote:
Regarding one complication: "The kernel needs to know where to allocate
a PROT_MTE page from or migrate a current page if it becomes PROT_MTE
(mprotect()) and the range it is in does not support tagging.",
simplified handling would be if it's in a MIGRATE_CMA pageblock, it
doesn't support tagging. You have to migrate to a !CMA page (for
example, not specifying GFP_MOVABLE as a quick way to achieve that).
Okay, I now realize that this patch set effectively duplicates some CMA
behavior using a new migrate-type.
[...]
I considered mixing the tag storage memory memory with normal memory and
adding it to MIGRATE_CMA. But since tag storage memory cannot be tagged,
this means that it's not enough anymore to have a __GFP_MOVABLE allocation
request to use MIGRATE_CMA.
I considered two solutions to this problem:
1. Only allocate from MIGRATE_CMA is the requested memory is not tagged =>
this effectively means transforming all memory from MIGRATE_CMA into the
MIGRATE_METADATA migratetype that the series introduces. Not very
appealing, because that means treating normal memory that is also on the
MIGRATE_CMA lists as tagged memory.
That's indeed not ideal. We could try this if it makes the patches
significantly simpler, though I'm not so sure.
Allocating metadata is the easier part as we know the correspondence
from the tagged pages (32 PROT_MTE page) to the metadata page (1 tag
storage page), so alloc_contig_range() does this for us. Just adding it
to the CMA range is sufficient.
However, making sure that we don't allocate PROT_MTE pages from the
metadata range is what led us to another migrate type. I guess we could
achieve something similar with a new zone or a CPU-less NUMA node,
Ideally, no significant core-mm changes to optimize for an architecture
oddity. That implies, no new zones and no new migratetypes -- unless it
is unavoidable and you are confident that you can convince core-MM
people that the use case (giving back 3% of system RAM at max in some
setups) is worth the trouble.
I also had CPU-less NUMA nodes in mind when thinking about that, but not
sure how easy it would be to integrate it. If the tag memory has
actually different performance characteristics as well, a NUMA node
would be the right choice.
If we could find some way to easily support this either via CMA or
CPU-less NUMA nodes, that would be much preferable; even if we cannot
cover each and every future use case right now. I expect some issues
with CXL+MTE either way , but are happy to be taught otherwise :)
Another thought I had was adding something like CMA memory
characteristics. Like, asking if a given CMA area/page supports tagging
(i.e., flag for the CMA area set?)?
When you need memory that supports tagging and have a page that does not
support tagging (CMA && taggable), simply migrate to !MOVABLE memory
(eventually we could also try adding !CMA).
Was that discussed and what would be the challenges with that? Page
migration due to compaction comes to mind, but it might also be easy to
handle if we can just avoid CMA memory for that.
though the latter is not guaranteed not to allocate memory from the
range, only make it less likely. Both these options are less flexible in
terms of size/alignment/placement.
Maybe as a quick hack - only allow PROT_MTE from ZONE_NORMAL and
configure the metadata range in ZONE_MOVABLE but at some point I'd
expect some CXL-attached memory to support MTE with additional carveout
reserved.
I have no idea how we could possibly cleanly support memory hotplug in
virtual environments (virtual DIMMs, virtio-mem) with MTE. In contrast
to s390x storage keys, the approach that arm64 with MTE took here
(exposing tag memory to the VM) makes it rather hard and complicated.
--
Cheers,
David / dhildenb
