On Mon, Nov 13, 2023 at 12:45 AM Casey Schaufler <casey@xxxxxxxxxxxxxxxx> wrote: > > On 11/11/2023 11:34 PM, Yafang Shao wrote: > > Background > > ========== > > > > In our containerized environment, we've identified unexpected OOM events > > where the OOM-killer terminates tasks despite having ample free memory. > > This anomaly is traced back to tasks within a container using mbind(2) to > > bind memory to a specific NUMA node. When the allocated memory on this node > > is exhausted, the OOM-killer, prioritizing tasks based on oom_score, > > indiscriminately kills tasks. This becomes more critical with guaranteed > > tasks (oom_score_adj: -998) aggravating the issue. > > Is there some reason why you can't fix the callers of mbind(2)? > This looks like an user space configuration error rather than a > system security issue. It appears my initial description may have caused confusion. In this scenario, the caller is an unprivileged user lacking any capabilities. While a privileged user, such as root, experiencing this issue might indicate a user space configuration error, the concerning aspect is the potential for an unprivileged user to disrupt the system easily. If this is perceived as a misconfiguration, the question arises: What is the correct configuration to prevent an unprivileged user from utilizing mbind(2)?" > > > > > The selected victim might not have allocated memory on the same NUMA node, > > rendering the killing ineffective. This patch aims to address this by > > disabling MPOL_BIND in container environments. > > > > In the container environment, our aim is to consolidate memory resource > > control under the management of kubelet. If users express a preference for > > binding their memory to a specific NUMA node, we encourage the adoption of > > a standardized approach. Specifically, we recommend configuring this memory > > policy through kubelet using cpuset.mems in the cpuset controller, rather > > than individual users setting it autonomously. This centralized approach > > ensures that NUMA nodes are globally managed through kubelet, promoting > > consistency and facilitating streamlined administration of memory resources > > across the entire containerized environment. > > Changing system behavior for a single use case doesn't seem prudent. > You're introducing a bunch of kernel code to avoid fixing a broken > user space configuration. Currently, there is no mechanism in place to proactively prevent an unprivileged user from utilizing mbind(2). The approach adopted is to monitor mbind(2) through a BPF program and trigger an alert if its usage is detected. However, beyond this monitoring, the only recourse is to verbally communicate with the user, advising against the use of mbind(2). As a result, users will question why mbind(2) isn't outright prohibited in the first place. -- Regards Yafang
