Hey All, I'm sending this out before the v6 of "mm: userspace hugepage collapse" for the purposes of aligning on and finalizing the semantics of the proposed MADV_COLLAPSE madvise(2) mode. Background: So far, thanks to everyone's input, we've aligned on: - MADV_COLLAPSE specifies its own hugepage allocation semantics (it allows direct reclaim/compaction). - MADV_COLLAPSE ignores khugepaged heuristics (/sys/kernel/mm/transparent_hugepage/khugepaged/max_pte_* and young/referenced page requirements). In terms of THP _eligibility_, in v5 it was proposed that MADV_COLLAPSE follow existing THP eligibility semantics (/sys/kernel/mm/transparent_hugepage/enabled + the VMA flags of the VMA being collapsed)[1]. However, Rongwei Wang kindly pointed out that the useability of process_madvise(MADV_COLLAPSE) on a system in "madvise" THP mode was limited. I agreed to include process_madvise(2) support for MADV_[NO]HUGEPAGE in v6, but following a discussion with David H., I think that was a mistake. Namely, as David kindly pointed out, there exist programs that don't work with THP and have good reason to disable it. The example provided was postcopy life migration in QEMU, which explicitly disables THP right before faulting in any pages. Idea: MADV_COLLAPSE should respect VM_NOHUGEPAGE and "never" THP mode, but otherwise would attempt to collapse. Why? If someone(*), somewhere told us not to use THPs, then don't override that decision. Otherwise, this is an explicit, safe(**) request made on behalf of ourselves, or by a CAP_SYS_ADMIN process, and shouldn't be blocked by interfaces meant to guide the "transparent" part of THPs. Other options considered: I considered variations of setting VM_HUGEPAGE only if calling on behalf of self or if VM_NOHUGEPAGE is not set. However, I didn't like this because there isn't a way to undo the operation: If we supported process_madvise(MADV_NOHUGEPAGE), we would have to let the application unclear VM_NOHUGEPAGE because outside processes can't/shouldn't. It would have to require some *new* madvise mode like MADV_CLEARHUGEPAGE (that would fail if calling on behalf of another process and VM_NOHUGEPAGE set) to clear VM_[NO]HUGEPAGE. A possible downside to the proposed approach is that, if in "madvise" THP mode and collapsing a VMA not marked VM_HUGEPAGE, it's now the caller's responsibility to monitor and recollapse this memory back into THPs. However, in practice this likely means an explicit MADV_DONTNEED (please let me know if there are other important cases here), and presumably it's the caller's job to do the monitoring anyway. Thanks again for taking the time to read / provide input here. I think this is the last point to clear up before releasing a v6 that should hopefully have all the functionality we need. Best, Zach --- (*) If we could verify that "never" THP mode was used _only_ for debugging, then I'd actually opt to ignore "never" in MADV_COLLAPSE. It's the last dependency MADV_COLLAPSE has on sysfs THP interface and would provide a convenient way to test/debug MADV_COLLAPSE with khugepaged / at-fault disabled. (**) I suppose there could exist applications that see THP "madvise" mode, never call MADV_HUGEPAGE, and so assume THPs will never be found. [1] https://lore.kernel.org/linux-mm/20220504214437.2850685-1-zokeefe@xxxxxxxxxx/ [2] https://lore.kernel.org/linux-mm/502a3ced-f3c6-7117-3b24-d80d204d66ee@xxxxxxxxxxxxxxxxx/
