On Thu, Apr 21, 2022 at 02:46:00PM -0400, Waiman Long wrote: > On 4/21/22 13:59, Roman Gushchin wrote: > > On Thu, Apr 21, 2022 at 01:28:20PM -0400, Waiman Long wrote: > > > On 4/21/22 12:33, Roman Gushchin wrote: > > > > On Thu, Apr 21, 2022 at 10:58:45AM -0400, Waiman Long wrote: > > > > > For systems with large number of CPUs, the majority of the memory > > > > > consumed by the mem_cgroup structure is actually the percpu stats > > > > > memory. When a large number of memory cgroups are continuously created > > > > > and destroyed (like in a container host), it is possible that more > > > > > and more mem_cgroup structures remained in the dying state holding up > > > > > increasing amount of percpu memory. > > > > > > > > > > We can't free up the memory of the dying mem_cgroup structure due to > > > > > active references in some other places. However, the percpu stats memory > > > > > allocated to that mem_cgroup is a different story. > > > > > > > > > > This patch adds a new percpu_stats_disabled variable to keep track of > > > > > the state of the percpu stats memory. If the variable is set, percpu > > > > > stats update will be disabled for that particular memcg. All the stats > > > > > update will be forward to its parent instead. Reading of the its percpu > > > > > stats will return 0. > > > > > > > > > > The flushing and freeing of the percpu stats memory is a multi-step > > > > > process. The percpu_stats_disabled variable is set when the memcg is > > > > > being set to offline state. After a grace period with the help of RCU, > > > > > the percpu stats data are flushed and then freed. > > > > > > > > > > This will greatly reduce the amount of memory held up by dying memory > > > > > cgroups. > > > > > > > > > > By running a simple management tool for container 2000 times per test > > > > > run, below are the results of increases of percpu memory (as reported > > > > > in /proc/meminfo) and nr_dying_descendants in root's cgroup.stat. > > > > Hi Waiman! > > > > > > > > I've been proposing the same idea some time ago: > > > > https://lore.kernel.org/all/20190312223404.28665-7-guro@xxxxxx/T/ . > > > > > > > > However I dropped it with the thinking that with many other fixes > > > > preventing the accumulation of the dying cgroups it's not worth the added > > > > complexity and a potential cpu overhead. > > > > > > > > I think it ultimately comes to the number of dying cgroups. If it's low, > > > > memory savings are not worth the cpu overhead. If it's high, they are. > > > > I hope long-term to drive it down significantly (with lru-pages reparenting > > > > being the first major milestone), but it might take a while. > > > > > > > > I don't have a strong opinion either way, just want to dump my thoughts > > > > on this. > > > I have quite a number of customer cases complaining about increasing percpu > > > memory usages. The number of dying memcg's can go to tens of thousands. From > > > my own investigation, I believe that those dying memcg's are not freed > > > because they are pinned down by references in the page structure. I am aware > > > that we support the use of objcg in the page structure which will allow easy > > > reparenting, but most pages don't do that and it is not easy to do this > > > conversion and it may take quite a while to do that. > > The big question is whether there is a memory pressure on those systems. > > If yes, and the number of dying cgroups is growing, it's worth investigating. > > It might be due to the sharing of pagecache pages and this will be ultimately > > fixed with implementing of the pagecache reparenting. But it also might be due > > to other bugs, which are fixable, so it would be great to understand. > > > Pagecache reparenting will probably fix the problem that I have seen. Is > someone working on this? > We also encountered dying cgroup issue on our servers for a long time. I have worked on this for a while and proposed a resolution [1] based on obj_cgroup APIs to charge the LRU pages. [1] https://lore.kernel.org/all/20220216115132.52602-1-songmuchun@xxxxxxxxxxxxx/ Thanks.
