On Thu, Feb 06, 2025 at 06:50:40PM +0000, Frank van der Linden wrote: > v3: > * Fix SPDX comment include file format. > * Add new hugetlb_cma.* files to MAINTAINERS > * Document new ranges/ subdir in CMA debugfs. > * Fix powerpc compilation for config without HAVE_BOOTMEM_INFO_NODE > * Fix various other nits found by kernel test robot. > * Use a PFN value of -1 to indicate a non-mirrored mapping > in sparse-vmemmap.c, not 0. > * Fix incorrect if() statement that got mangled in cma.c > > v2: > * Add missing CMA debugfs code. > * Minor cleanups in hugetlb_cma changes. > * Move hugetlb_cma code to its own file to further clean > things up. > > On large systems, we observed some issues with hugetlb and CMA: > > 1) When specifying a large number of hugetlb boot pages (hugepages= > on the commandline), the kernel may run out of memory before it > even gets to HVO. For example, if you have a 3072G system, and > want to use 3024 1G hugetlb pages for VMs, that should leave > you plenty of space for the hypervisor, provided you have the > hugetlb vmemmap optimization (HVO) enabled. However, since > the vmemmap pages are always allocated first, and then later > in boot freed, you will actually run yourself out of memory > before you can do HVO. This means not getting all the hugetlb > pages you want, and worse, failure to boot if there is an > allocation failure in the system from which it can't recover. > > 2) There is a system setup where you might want to use hugetlb_cma > with a large value (say, again, 3024 out of 3072G like above), > and then lower that if system usage allows it, to make room > for non-hugetlb processes. For this, a variation of the problem > above applies: the kernel runs out of unmovable space to allocate > from before you finish boot, since your CMA area takes up all > the space. > > 3) CMA wants to use one big contiguous area for allocations. Which > fails if you have the aforementioned 3T system with a gap in the > middle of physical memory (like the < 40bits BIOS DMA area seen on > some AMD systems). You then won't be able to set up a CMA area for > one of the NUMA nodes, leading to loss of half of your hugetlb > CMA area. > > 4) Under the scenario mentioned in 2), when trying to grow the > number of hugetlb pages after dropping it for a while, new > CMA allocations may fail occasionally. This is not unexpected, > some transient references on pages may prevent cma_alloc > from succeeding under memory pressure. However, the hugetlb > code then falls back to a normal contiguous alloc, which may > end up succeeding. This is not always desired behavior. If > you have a large CMA area, then the kernel has a restricted > amount of memory it can do unmovable allocations from (a well > known issue). A normal contiguous alloc may eat further in to > this space. Hi Frank, While I plan to keep reviewing the series, I think it would make sense to split this patchset into two smaller ones. The way I see it, we are trying to deal with two different problems and their solutions. 1) pre-hvo at boot time 2) multi-range support of CMA (only used for hugetlb) I did not go through the entire patchset yet, so I ignore whether the respective patches to tackle these two problems are really dependent on each other, but I think that would be very interesting to consider a patchset per solution if that is not the case. IMHO, it would ease review quite a lot. -- Oscar Salvador SUSE Labs
