[...]
I dislike exposing "orders" to the users, I'm happy to be convinced why I am
wrong and it is a good idea.
So maybe "Small THP"/"Small-sized THP" is better. Or "Medium-sized THP" -- as
said, I think FreeBSD tends to call it "Medium-sized superpages". But what's
small/medium/large is debatable. "Small" implies at least that it's smaller than
what we used to know, which is a fact.
Can we also now use the terminology consistently? (e.g., "variable-order, large
folios for anonymous memory" -> "Small-sized anonymous THP", you can just point
at the previous patch set name in the cover letter)
Yes absolutely. FWIW, I was deliberately not changing the title of the patchset
so people could easily see it was an evolution of something posted before. But
if it's the norm to change the title as the patchset evolves, I'm very happy to
do that. And there are other places too, in commit logs that I can tidy up. I
will assume "PMD-sized THP", "small-sized THP" and "anonymous small-sized THP"
(that last one slightly different from what David suggested above - it means
"small-sized THP" can still be grepped) unless others object.
Absolutely fine with me. Hoping other people will object when I talk
nonsense or my suggestions don't make any sense.
Or even better, propose something better :)
1) Since SW (the kernel) is dealing with larger chunks of memory than base
pages, there are efficiency savings to be had; fewer page faults, batched PTE
and RMAP manipulation, reduced lru list, etc. In short, we reduce kernel
overhead. This should benefit all architectures.
2) Since we are now mapping physically contiguous chunks of memory, we can take
advantage of HW TLB compression techniques. A reduction in TLB pressure
speeds up kernel and user space. arm64 systems have 2 mechanisms to coalesce
TLB entries; "the contiguous bit" (architectural) and HPA (uarch).
The major change in this revision is the addition of sysfs controls to allow
this "small-order THP" to be enabled/disabled/configured independently of
PMD-order THP. The approach I've taken differs a bit from previous discussions;
instead of creating a whole new interface ("large_folio"), I'm extending THP. I
personally think this makes things clearer and more extensible. See [6] for
detailed rationale.
Change 2: sysfs interface.
If we call it THP, it shall go under "/sys/kernel/mm/transparent_hugepage/", I
agree.
What we expose there and how, is TBD. Again, not a friend of "orders" and
bitmaps at all. We can do better if we want to go down that path.
Maybe we should take a look at hugetlb, and how they added support for multiple
sizes. What *might* make sense could be (depending on which values we actually
support!)
/sys/kernel/mm/transparent_hugepage/hugepages-64kB/
/sys/kernel/mm/transparent_hugepage/hugepages-128kB/
/sys/kernel/mm/transparent_hugepage/hugepages-256kB/
/sys/kernel/mm/transparent_hugepage/hugepages-512kB/
/sys/kernel/mm/transparent_hugepage/hugepages-1024kB/
/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/
Each one would contain an "enabled" and "defrag" file. We want something minimal
first? Start with the "enabled" option.
enabled: always [global] madvise never
Initially, we would set it for PMD-sized THP to "global" and for everything else
to "never".
My only reservation about this approach is the potential for a future need for a
"one setting applied across all sizes" class of control (e.g. "auto"). I think
we agreed in the previous meetings that chasing a solution for "auto" was a good
aspiration to have, so it would be good to have a place we we can insert that in
future. The main reason why I chose to expose the "anon_orders" control is
because it is possible to both enable/disable the various sizes as well as
specificy (e.g.) "auto", without creating redundancy. But I agree that ideally
we wouldn't expose orders to the user; I was attempting a compromise to simplify
the "auto" case.
A potential (though feels quite complex) solution to make auto work with your
proposal: Add "auto" as an option to the existing global enabled file, and to
all of your proposed new enabled files. But its only possible to *set* auto
through the global file. And when it is set, all of the size-specific enabled
files read-back "auto" too. Any any writes to the size-specific enabled files
are ignored (or remembered but not enacted) until the global enabled file is
changed away from auto.
Yes, I think there are various ways forward regarding that. Or to enable
"auto" mode only once all are "auto", and as soon as one is not "auto",
just disable it. A simple
echo "auto" > /sys/kernel/mm/transparent_hugepage/hugepages-*/enabled
Would do to enable it. Or, have them all be "global" and have a global
"auto" mode as you raised.
echo "global" > /sys/kernel/mm/transparent_hugepage/hugepages-*/enabled
echo "auto" > /sys/kernel/mm/transparent_hugepage/enabled
But I'm not sure if adding a new option to the global enabled file might break
compat?
I think we used to extend the "defrag" option, see
commit 21440d7eb9044001b7fdb71d0163689f60a0f2a1
Author: David Rientjes <rientjes@xxxxxxxxxx>
Date: Wed Feb 22 15:45:49 2017 -0800
mm, thp: add new defer+madvise defrag option
So I suspect we could extend that one in a similar way.
But again, this is just the thing that came to mind when thinking about
how to:
a) avoid orders
b) make it configurable and future-proof
c) make it look a bit consistent with other interfaces (hugetlb and
existing thp)
d) still prepare for an auto mode that we want in the future
I'm happy to hear other ideas.
That sounds reasonable at least to me, and we would be using what we learned
from THP (as John suggested). That still gives reasonable flexibility without
going too wild, and a better IMHO interface.
I understand Yu's point about ABI discussions and "0 knobs". I'm happy as long
as we can have something that won't hurt us later and still be able to use this
in distributions within a reasonable timeframe. Enabling/disabling individual
sizes does not sound too restrictive to me. And we could always add an "auto"
setting later and default to that with a new kconfig knob.
If someone wants to configure it, why not. Let's just prepare a way to to handle
this "better" automatically in the future (if ever ...).
Change 3: Stats
/proc/meminfo:
Introduce new "AnonHugePteMap" field, which reports the amount of
memory (in KiB) mapped from large folios globally (similar to
AnonHugePages field).
AnonHugePages is and remains "PMD-sized THP that is mapped using a PMD", I think
we all agree on that. It should have been named "AnonPmdMapped" or
"AnonHugePmdMapped", too bad, we can't change that.
Yes agreed. I did consider redefining "AnonHugePages" to cover PMD- and
PTE-mapped memory, then introduce both an "AnonHugePmdMapped" and
"AnonHugePteMapped", but I think that would likely break things. Its further
complicated because vmstats prints it in PMD-size units, so can't represent
PTE-mapped memory in that counter.
:/
"AnonHugePteMap" better be "AnonHugePteMapped".
I agree, but I went with the shorter one because any longer and it would unalign
the value e.g:
AnonHugePages: 0 kB
AnonHugePteMapped: 0 kB
ShmemPmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Can't that be handled? We surely have long stuff in there:
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 1081344 kB
ShmemPmdMapped: 0 kB
HardwareCorrupted has same length as AnonHugePteMapped
But I'm not convinced about "AnonHugePteMapped" yet :)
So would need to decide which is preferable, or come up with a shorter name.
But, I wonder if we want to expose this "PteMapped" to user space *at all*. Why
should they care if it's PTE mapped? For PMD-sized THP it makes a bit of sense,
because !PMD implied !performance, and one might have been able to troubleshoot
that somehow. For PTE-mapped, it doesn't make much sense really, they are always
PTE-mapped.
I disagree; I've been using it a lot to debug performance issues. It tells you
how much of your anon memory is allocated with large folios. And making that
percentage bigger improves performance; fewer page faults, and with a separate
contpte series on arm64, better use of the TLB. Reasons might include; poorly
aligned/too small VMAs, memory fragmentation preventing allocation, CoW, etc.
Just because a small-sized THP is PTE-mapped doesn't tell you anything,
really. What you want to know is if it is "completely" and
"consecutively" mapped such that the HW can actually benefit from it --
if HW even supports it. So "PTE-mapped THP" is just part of the story.
And that's where it gets tricky I think.
I agree that it's good for debugging, but then maybe it should a) live
somewhere else (debugfs, bucketing below) and b) be consistent with
other THPs, meaning we also want similar stats somewhere.
One idea would be to expose such stats in a R/O fashion like
"nr_allocated" or "nr_hugepages" in
/sys/kernel/mm/transparent_hugepage/hugepages-64kB/ and friends. Of
course, maybe tagging them with "anon" prefix.
I would actually argue for adding similar counters for file-backed memory too
for the same reasons. (I actually posted an independent patch a while back that
did this for file- and anon- memory, bucketted by size. But I think the idea of
the bucketting was NAKed.
For debugging, I *think* it might be valuable to see how many THP of
each size are allocated. Tracking exactly "how is it mapped" is not easy
to achieve as we learned. PMD-mapped was easy, but also requires us to
keep doing that tracking for all eternity ...
Do you have a pointer to the patch set? Did it try to squeeze it into
/proc/meminfo?
That also raises the question how you would account a PTE-mapped THP. The hole
thing? Only the parts that are mapped? Let's better not go down that path.
The approach I've taken in this series is the simple one - account every page
that belongs to a large folio from when it is first mapped to last unmapped.
Yes, in this case, you might not actually be mapping the full thing
contigiously. But it gives a good indication.
I also considered accounting the whole folio only when all of its pages become
mapped (although not worrying about them all being contiguous). That's still
simple to implement for all counters except smaps. So went with the simplest
approach with the view that its "good enough".
If you take a look at "ShmemHugePages" and "FileHugePages", there we
actually track them when they get allocated+freed, which is much easier
than tracking when/how they are (un)mapped. But it's only done for
PMD-sized THP for now.
That leaves the question why we would want to include them here at all in a
special PTE-mapped way?
Again, let's look at hugetlb: I prepared 1 GiB and one 2 MiB page.
HugePages_Total: 1
HugePages_Free: 1
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 1050624 kB
-> Only the last one gives the sum, the other stats don't even mention the other
ones. [how do we get their stats, if at all?]
There are some files in /sys/kernel/mm/hugepages/hugepages-XXkB and
/sys/devices/system/node/node*/hugepages/; nr_hugepages, free_hugepages,
surplus_hugepages. But this interface also constitutes the allocator, not just
stats, I think.
Ah, I missed that we expose free vs. reserved vs. surpluse ... there as
well; I thought we would only have "nr_hugepages".
So maybe, we only want a summary of how many anon huge pages of any size are
allocated (independent of the PTE vs. PMD mapping),
Are you proposing (AnonHugePages + AnonHugePteMapped) here or something else? If
the former, then I don't really see the difference. We have to continue to
expose PMD-size (AnonHugePages). So either add PTE-only counter, and derive the
total, or add a total counter and derive PTE-only. I suspect I've misunderstood
your point.
I don't think we should go down the "PteMapped" path. Probably we want
"bucketing" stats as you said, and maybe a global one that just combines
everything (any THP). But naming will be difficult.
and some other source to
eventually inspect how the different sizes behave.
But note that for non-PMD-sized file THP we don't even have special counters!
... so maybe we should also defer any such stats and come up with something
uniform for all types of non-PMD-sized THP.
Indeed, I can see benefit in adding these for file THP - in fact I have a patch
that does exactly that to help my development work. I had envisaged that we
could add something like FileHugePteMapped, ShmemHugePteMapped that would follow
the same semantics as AnonHugePteMapped.
Again, maybe we can find something that does not involve the "PteMapped"
terminology and just gives us a big total of "allocated" THP. For
detailed stats for debugging, maybe we can just use a different
interface then.
Sane discussion applies to all other stats.
Because we now have runtime enable/disable control, I've removed the compile
time Kconfig switch. It still defaults to runtime-disabled.
NOTE: These changes should not be merged until the prerequisites are complete.
These are in progress and tracked at [7].
We should probably list them here, and classify which one we see as strict a
requirement, which ones might be an optimization.
I'll need some help with clasifying them, so showing my working. Final list that
I would propose as strict requirements at bottom.
This is my list with status, as per response to Yu in other thread:
- David is working on "shared vs exclusive mappings"
Probably "COW reuse support" is a separate item, although my approach
would cover that.
The question is, if the estimate we're using in most code for now would
at least be sufficient to merge it. The estimate is easily wrong, but we
do have that issue with PTE-mapped THP already.
But that argument probably applies to most things here: the difference
is that PTE-mapped THP are not the default, that's why nobody really cared.
[I'm playing with an almost-lockless scheme right now and hope I have
something running soonish -- as you know, I got distracted]
- Zi Yan has posted an RFC for compaction
- Yin Fengwei's mlock series is now in mm-stable
- Yin Fengwei's madvise series is in 6.6
- I've reworked and posted a series for deferred_split_folio; although I've
deprioritied it because Yu said it wasn't really a pre-requisite.
- numa balancing depends on David's "shared vs exclusive mappings" work
- I've started looking at "large folios in swap cache" in the background,
because I'm seeing some slow down with large folios, but we also agreed that
wasn't a prerequisite
Probably it would be good to talk about the items and how we would
classify them in a meeting.
Although, since sending that, I've determined that when running kernel
compilation across high number of cores on arm64, the cost of splitting the
folios gets large due to needing to broadcast the extra TLBIs. So I think the
last point on that list may be a prerequisite after all. (I've been able to fix
this by adding support for allocating large folios in the swap file, and
avoiding the split - planning to send RFC this week).
There is also this set of things that you mentioned against "shared vs exclusive
mappings", which I'm not sure if you are planning to cover as part of your work
or if they are follow on things that will need to be done:
(1) Detecting shared folios, to not mess with them while they are shared.
MADV_PAGEOUT, user-triggered page migration, NUMA hinting, khugepaged ...
replace cases where folio_estimated_sharers() == 1 would currently be the
best we can do (and in some cases, page_mapcount() == 1).
And I recently discovered that khugepaged doesn't collapse file-backed pages to
a PMD-size THP if they belong to a large folio, so I'm guessing it may also
suffer the same behaviour for anon memory. I'm not sure if that's what your
"khugepaged ..." comment refers to?
Yes. But I did not look into all the details yet.
"kuhepaged" collapse support to small-sized THP is probably also a very
imporant item, although it might be less relevant than for PMD -- and I
consider it future work. See below.
So taking all that and trying to put together a complete outstanding list for
strict requirements:
- Shared vs Exclusive Mappings (DavidH)
- user-triggered page migration
- NUMA hinting/balancing
- Enhance khugepaged to collapse to PMD-size from PTE-mapped large folios
- Compaction of Large Folios (Zi Yan)
- Swap out small-size THP without Split (Ryan Roberts)
^ that's going to be tough, I can promise. And the only way to live
without that would be khugepaged support. (because that's how it's all
working for PMD-sized THP after all!)
Once a PMD-sized THP was swapped out and evicted, it will always come
back in order-0 folios. khugeged will re-collapse into PMD-sized chunks.
If we could do that for PTE-sized THP as well ...
Now, these are just my thoughts, and I'm happy about other thoughts.
As always, thanks for taking the time - I really appreciate it.
Sure. Hoping others can comment.
My gut feeling is that it's best to focus on getting the sysfs interface
right+future proof and handling the stats independently. While being a
good debug mechanism, I wouldn't consider these stats a requirement: we
don't have them for file/shmem small-sized thp so far as well.
So maybe really better to handle the stats in meminfo and friends
separately.
--
Cheers,
David / dhildenb
