Topic: Memory cgroups, whether you like it or not 1. Memory cgroup counters scalability Recently, benchmark teams at Intel were running some bare-metal benchmarks. To our great surprise, we saw lots of memcg activity in the profiles. When we asked the benchmark team, they did not even realize they were using memory cgroups. They were fond of running all their benchmarks in containers that just happened to use memory cgroups by default. What were previously problems only for memory cgroup users are quickly becoming a problem for everyone. There are mem cgroup counters that are read in page management paths which scale poorly when read. These counters are per cpu based and need to be summed over all CPUs to get the overall value for the mem cgroup in lruvec_page_state_local function. This led to scalability problems on system with large numbers of CPUs. For example, we’ve seen 14+% kernel CPU time consumed in snapshot_refaults(). We have also encountered a similar issue recently when computing the lru_size[1]. We'll like to do some brainstorming to see if there are ways to make such accounting more scalable. For example, not all usages of such counters need precise counts, and some approximate counts that are updated lazily can be used. [1] https://lore.kernel.org/linux-mm/a64eecf1-81d4-371f-ff6d-1cb057bd091c@xxxxxxxxxxxxxxx/ 2. Tiered memory accounting and management Traditionally, all RAM is DRAM. Some DRAM might be closer/faster than others, but a byte of media has about the same cost whether it is close or far. But, with new memory tiers such as High-Bandwidth Memory or Persistent Memory, there is a choice between fast/expensive and slow/cheap. But, the current memory cgroups still live in the old model. There is only one set of limits, and it implies that all memory has the same cost. We would like to extend memory cgroups to comprehend different memory tiers to give users a way to choose a mix between fast/expensive and slow/cheap. We would like to propose that for systems with multiple memory tiers, We will add accounting per mem cgroup for a memory cgroup's usage of the top tier memory. Such top tier memory are precious resources, where it makes sense to impose soft limits. We can start to actively demote the top tier memory used by cgroups that exceed their allowance when the system experience memory pressure in the top tier memory. There is existing infrastructure for memory soft limit per cgroup we can leverage to implement such a scheme. We'll like to find out if this approach makes sense for people working on systems with multiple memory tiers.
