On Mon, May 23, 2022 at 5:19 PM Zach O'Keefe <zokeefe@xxxxxxxxxx> wrote: > > Hey All, > > I'm sending this out before the v6 of "mm: userspace hugepage > collapse" for the purposes of aligning on and finalizing the semantics > of the proposed MADV_COLLAPSE madvise(2) mode. > > Background: > > So far, thanks to everyone's input, we've aligned on: > - MADV_COLLAPSE specifies its own hugepage allocation semantics (it > allows direct reclaim/compaction). > - MADV_COLLAPSE ignores khugepaged heuristics > (/sys/kernel/mm/transparent_hugepage/khugepaged/max_pte_* and > young/referenced page requirements). > > In terms of THP _eligibility_, in v5 it was proposed that > MADV_COLLAPSE follow existing THP eligibility semantics > (/sys/kernel/mm/transparent_hugepage/enabled + the VMA flags of the > VMA being collapsed)[1]. > > However, Rongwei Wang kindly pointed out that the useability of > process_madvise(MADV_COLLAPSE) on a system in "madvise" THP mode was > limited. I agreed to include process_madvise(2) support for > MADV_[NO]HUGEPAGE in v6, but following a discussion with David H., I > think that was a mistake. Namely, as David kindly pointed out, there > exist programs that don't > work with THP and have good reason to disable it. The example > provided was postcopy life migration in QEMU, which explicitly > disables THP right before faulting in any pages. > > Idea: MADV_COLLAPSE should respect VM_NOHUGEPAGE and "never" THP mode, > but otherwise would attempt to collapse. I do agree to respect VM_NOHUGEPAGE and "never". Collapsing for not-madvised VMAs for "madvise" mode sounds ok to me, but I'm not so sure. > > Why? If someone(*), somewhere told us not to use THPs, then don't > override that decision. Otherwise, this is an explicit, safe(**) > request made on behalf of ourselves, or by a CAP_SYS_ADMIN process, > and shouldn't be blocked by interfaces meant to guide the > "transparent" part of THPs. > > Other options considered: > > I considered variations of setting VM_HUGEPAGE only if calling on > behalf of self or if VM_NOHUGEPAGE is not set. However, I didn't like > this because there isn't a way to undo the operation: If we supported > process_madvise(MADV_NOHUGEPAGE), we would have to let the application > unclear VM_NOHUGEPAGE because outside processes can't/shouldn't. It > would have to require some *new* madvise mode like MADV_CLEARHUGEPAGE > (that would fail if calling on behalf of another process and > VM_NOHUGEPAGE set) to clear VM_[NO]HUGEPAGE. > > A possible downside to the proposed approach is that, if in "madvise" > THP mode and collapsing a VMA not marked VM_HUGEPAGE, it's now the > caller's responsibility to monitor and recollapse this memory back > into THPs. However, in practice this likely means an explicit > MADV_DONTNEED (please let me know if there are other important cases > here), and presumably it's the caller's job to do the monitoring anyway. Page reclaim could also cause the THP split. And it may happen at any time. I'm not sure how the users or callers could monitor it. > > Thanks again for taking the time to read / provide input here. I think > this is the last point to clear up before releasing a v6 that should > hopefully have all the functionality we need. > > Best, > Zach > > --- > > (*) If we could verify that "never" THP mode was used _only_ for > debugging, then I'd actually opt to ignore "never" in MADV_COLLAPSE. > It's the last dependency MADV_COLLAPSE has on sysfs THP interface and > would provide a convenient way to test/debug MADV_COLLAPSE with > khugepaged / at-fault disabled. > (**) I suppose there could exist applications that see THP "madvise" > mode, never call MADV_HUGEPAGE, and so assume THPs will never be > found. > > [1] https://lore.kernel.org/linux-mm/20220504214437.2850685-1-zokeefe@xxxxxxxxxx/ > [2] https://lore.kernel.org/linux-mm/502a3ced-f3c6-7117-3b24-d80d204d66ee@xxxxxxxxxxxxxxxxx/
