This series can minimize the overhead of struct page for 2MB HugeTLB pages
significantly. It further reduces the overhead of struct page by 12.5% for
a 2MB HugeTLB compared to the previous approach, which means 2GB per 1TB
HugeTLB. It is a nice gain. Comments and reviews are welcome. Thanks.
The main implementation and details can refer to the commit log of patch 1.
In this series, I have changed the following four helpers, the following
table shows the impact of the overhead of those helpers.
+------------------+-----------------------+
| APIs | head page | tail page |
+------------------+-----------+-----------+
| PageHead() | Y | N |
+------------------+-----------+-----------+
| PageTail() | Y | N |
+------------------+-----------+-----------+
| PageCompound() | N | N |
+------------------+-----------+-----------+
| compound_head() | Y | N |
+------------------+-----------+-----------+
Y: Overhead is increased.
N: Overhead is _NOT_ increased.
It shows that the overhead of those helpers on a tail page don't change
between "hugetlb_free_vmemmap=on" and "hugetlb_free_vmemmap=off". But the
overhead on a head page will be increased when "hugetlb_free_vmemmap=on"
(except PageCompound()). So I believe that Matthew Wilcox's folio series
will help with this.
The users of PageHead() and PageTail() are much less than compound_head()
and most users of PageTail() are VM_BUG_ON(), so I have done some tests
about the overhead of compound_head() on head pages.
I have tested the overhead of calling compound_head() on a head page, which
is 2.11ns (Measure the call time of 10 million times compound_head(), and
then average).
For a head page whose address is not aligned with PAGE_SIZE or a non-compound
page, the overhead of compound_head() is 2.54ns which is increased by 20%.
For a head page whose address is aligned with PAGE_SIZE, the overhead of
compound_head() is 2.97ns which is increased by 40%. Most pages are the former.
I do not think the overhead is significant since the overhead of compound_head()
itself is low.
Changlogs in v7:
1. Fix a typo (change page_head_if_fake to page_fixed_fake_head) in the
commit log of patch 2.
2. Move details of implementation from cover letter to the commit log of
patch 1.
3. Add some overhead numbers to the cover letter.
Changlogs in v6:
1. Add test case to tools/testing/selftests/vm/run_vmtests.sh.
Changlogs in v5:
1. Move NR_RESET_STRUCT_PAGE to the front of reset_struct_pages().
2. Collect Reviewed-by tags.
Thanks Barry for his suggestions and reviews.
Changlogs in v4:
1. Move hugetlb_free_vmemmap_enabled from hugetlb.h to page-flags.h.
2. Collect Reviewed-by.
3. Add a new patch to move vmemmap functions related to HugeTLB to
the scope of the CONFIG_HUGETLB_PAGE_FREE_VMEMMAP.
Thanks Barry for his suggestions and reviews.
Changlogs in v3:
1. Rename page_head_if_fake() to page_fixed_fake_head().
2. Introducing a new helper page_is_fake_head() to make code more readable.
3. Update commit log of patch 3 to add more judgements.
4. Add some comments in check_page_flags() in the patch 4.
Thanks Barry for his suggestions and reviews.
Changlogs in v2:
1. Drop two patches of introducing PAGEFLAGS_MASK from this series.
2. Let page_head_if_fake() return page instead of NULL.
3. Add a selftest to check if PageHead or PageTail work well.
Muchun Song (5):
mm: hugetlb: free the 2nd vmemmap page associated with each HugeTLB
page
mm: hugetlb: replace hugetlb_free_vmemmap_enabled with a static_key
mm: sparsemem: use page table lock to protect kernel pmd operations
selftests: vm: add a hugetlb test case
mm: sparsemem: move vmemmap related to HugeTLB to
CONFIG_HUGETLB_PAGE_FREE_VMEMMAP
Documentation/admin-guide/kernel-parameters.txt | 2 +-
include/linux/hugetlb.h | 6 -
include/linux/mm.h | 2 +
include/linux/page-flags.h | 90 ++++++++++++++-
mm/hugetlb_vmemmap.c | 68 ++++++-----
mm/memory_hotplug.c | 2 +-
mm/ptdump.c | 16 ++-
mm/sparse-vmemmap.c | 70 +++++++++---
tools/testing/selftests/vm/.gitignore | 1 +
tools/testing/selftests/vm/Makefile | 1 +
tools/testing/selftests/vm/hugepage-vmemmap.c | 144 ++++++++++++++++++++++++
tools/testing/selftests/vm/run_vmtests.sh | 11 ++
12 files changed, 350 insertions(+), 63 deletions(-)
create mode 100644 tools/testing/selftests/vm/hugepage-vmemmap.c
--
2.11.0
