On Thu, Apr 29, 2021 at 7:32 AM Shakeel Butt <shakeelb@xxxxxxxxxx> wrote: > > On Wed, Apr 28, 2021 at 2:54 AM Muchun Song <songmuchun@xxxxxxxxxxxxx> wrote: > > > > In our server, we found a suspected memory leak problem. The kmalloc-32 > > consumes more than 6GB of memory. Other kmem_caches consume less than 2GB > > memory. > > > > After our in-depth analysis, the memory consumption of kmalloc-32 slab > > cache is the cause of list_lru_one allocation. > > > > crash> p memcg_nr_cache_ids > > memcg_nr_cache_ids = $2 = 24574 > > > > memcg_nr_cache_ids is very large and memory consumption of each list_lru > > can be calculated with the following formula. > > > > num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32) > > > > There are 4 numa nodes in our system, so each list_lru consumes ~3MB. > > > > crash> list super_blocks | wc -l > > 952 > > > > Every mount will register 2 list lrus, one is for inode, another is for > > dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3 > > MB (~5.6GB). But the number of memory cgroup is less than 500. So I > > guess more than 12286 containers have been deployed on this machine (I > > do not know why there are so many containers, it may be a user's bug or > > the user really want to do that). But now there are less than 500 > > containers in the system. And memcg_nr_cache_ids has not been reduced > > to a suitable value. This can waste a lot of memory. If we want to reduce > > memcg_nr_cache_ids, we have to reboot the server. This is not what we > > want. > > > > So this patchset will dynamically adjust the value of memcg_nr_cache_ids > > to keep healthy memory consumption. In this case, we may be able to restore > > a healthy environment even if the users have created tens of thousands of > > memory cgroups and then destroyed those memory cgroups. This patchset also > > contains some code simplification. > > > > There was a recent discussion [1] on the same issue. Did you get the > chance to take a look at that. I have not gone through this patch > series yet but will do in the next couple of weeks. > > [1] https://lore.kernel.org/linux-fsdevel/20210405054848.GA1077931@xxxxxxxxxx/ Thanks for your reminder. No, I haven't. But now I have looked at this. The issue is very similar to mine. But Bharata seems to want to run 10k containers. And optimize the memory consumption of list_lru_one in this case. This is not what I do. I want to try to shrink the size of the list lrus when the number of memcgs is reduced from tens of thousands to hundreds. Thanks.
