On 6/1/23 15:53, Johannes Weiner wrote:
On Thu, Jun 01, 2023 at 03:15:28PM -0400, Waiman Long wrote:
On 6/1/23 14:38, Dan Schatzberg wrote:
The existing documentation refers to memory.high as the "main mechanism
to control memory usage." This seems incorrect to me - memory.high can
result in reclaim pressure which simply leads to stalls unless some
external component observes and actions on it (e.g. systemd-oomd can be
used for this purpose). While this is feasible, users are unaware of
this interaction and are led to believe that memory.high alone is an
effective mechanism for limiting memory.
The documentation should recommend the use of memory.max as the
effective way to enforce memory limits - it triggers reclaim and results
in OOM kills by itself.
That is not how my understanding of memory.high works. When memory usage
goes past memory.high, memory reclaim will be initiated to reclaim the
memory back. Stall happens when memory.usage keep increasing like by
consuming memory faster than what memory reclaim can recover. When
memory.max is reached, OOM killer will then kill off the tasks.
This was the initial plan indeed: Slow down the workload and thus slow
the growth; hope that the workload recovers with voluntary frees; set
memory.max as a safety if it keeps going beyond.
This never panned out. Once workloads are stuck, they might not back
down on their own. By increasingly slowing growth, it becomes harder
and harder for them to reach the memory.max intervention point.
It's a very brittle configuration strategy. Unless you very carefully
calibrate memory.high and memory.max together with awareness of the
throttling algorithm, workloads that hit memory.high will just go to
sleep indefinitely. They require outside intervention that either
adjusts limits or implements kill policies based on observed sleeps
(they're reported as pressure via psi).
So the common usecases today end up being that memory.max is for
enforcing kernel OOM kills, and memory.high is a tool to implement
userspace OOM killing policies.
Dan is right to point out the additional expectations for userspace
management when memory.high is in used. And memory.max is still the
primary, works-out-of-the-box method of memory containment.
Thanks for clarification. I have to reset my false assumption.
Cheers,
Longman
