Background
==========
In our containerized environment, we've identified unexpected OOM events
where the OOM-killer terminates tasks despite having ample free memory.
This anomaly is traced back to tasks within a container using mbind(2) to
bind memory to a specific NUMA node. When the allocated memory on this node
is exhausted, the OOM-killer, prioritizing tasks based on oom_score,
indiscriminately kills tasks.
The Challenge
============
In a containerized environment, independent memory binding by a user can
lead to unexpected system issues or disrupt tasks being run by other users
on the same server. If a user genuinely requires memory binding, we will
allocate dedicated servers to them by leveraging kubelet deployment.
Currently, users possess the ability to autonomously bind their memory to
specific nodes without explicit agreement or authorization from our end.
It's imperative that we establish a method to prevent this behavior.
Proposed Solutions
=================
- Introduce Capability to Disable MPOL_BIND
Currently, any task can perform MPOL_BIND without specific capabilities.
Enforcing CAP_SYS_RESOURCE or CAP_SYS_NICE could be an option, but this
may have unintended consequences. Capabilities, being broad, might grant
unnecessary privileges. We should explore alternatives to prevent
unexpected side effects.
- Use LSM BPF to Disable MPOL_BIND
Introduce LSM hooks for syscalls such as mbind(2), set_mempolicy(2), and
set_mempolicy_home_node(2) to disable MPOL_BIND. This approach is more
flexibility and allows for fine-grained control without unintended
consequences. A sample LSM BPF program is included, demonstrating
practical implementation in a production environment.
- seccomp
seccomp is relatively heavyweight, making it less suitable for
enabling in our production environment:
- Both kubelet and containers need adaptation to support it.
- Dynamically altering security policies for individual containers
without interrupting their operations isn't straightforward.
Future Considerations
=====================
In addition, there's room for enhancement in the OOM-killer for cases
involving CONSTRAINT_MEMORY_POLICY. It would be more beneficial to
prioritize selecting a victim that has allocated memory on the same NUMA
node. My exploration on the lore led me to a proposal[0] related to this
matter, although consensus seems elusive at this point. Nevertheless,
delving into this specific topic is beyond the scope of the current
patchset.
[0] https://lore.kernel.org/lkml/20220512044634.63586-1-ligang.bdlg@xxxxxxxxxxxxx/
Changes:
- RFC v1 -> RFC v2:
- Refine the commit log to avoid misleading
- Use one common lsm hook instead and add comment for it
- Add selinux implementation
- Other improments in mempolicy
- RFC v1: https://lwn.net/Articles/951188/
Yafang Shao (6):
mm, doc: Add doc for MPOL_F_NUMA_BALANCING
mm: mempolicy: Revise comment regarding mempolicy mode flags
mm, security: Fix missed security_task_movememory() in mbind(2)
mm, security: Add lsm hook for memory policy adjustment
security: selinux: Implement set_mempolicy hook
selftests/bpf: Add selftests for set_mempolicy with a lsm prog
.../admin-guide/mm/numa_memory_policy.rst | 27 +++++++
include/linux/lsm_hook_defs.h | 3 +
include/linux/security.h | 9 +++
include/uapi/linux/mempolicy.h | 2 +-
mm/mempolicy.c | 17 +++-
security/security.c | 13 +++
security/selinux/hooks.c | 8 ++
security/selinux/include/classmap.h | 2 +-
tools/testing/selftests/bpf/Makefile | 2 +-
.../selftests/bpf/prog_tests/set_mempolicy.c | 79 +++++++++++++++++++
.../selftests/bpf/progs/test_set_mempolicy.c | 29 +++++++
11 files changed, 187 insertions(+), 4 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/set_mempolicy.c
create mode 100644 tools/testing/selftests/bpf/progs/test_set_mempolicy.c
--
2.30.1 (Apple Git-130)
