Hi Yuanchu, On Wed, 29 Jan 2025 18:02:26 -0800 Yuanchu Xie <yuanchu@xxxxxxxxxx> wrote: > On Wed, Dec 11, 2024 at 11:53 AM SeongJae Park <sj@xxxxxxxxxx> wrote: > > > > On Fri, 6 Dec 2024 11:57:55 -0800 Yuanchu Xie <yuanchu@xxxxxxxxxx> wrote: > > > > > Thanks for the response Johannes. Some replies inline. > > > > > > On Tue, Nov 26, 2024 at 11:26\u202fPM Johannes Weiner <hannes@xxxxxxxxxxx> wrote: > > > > > > > > On Tue, Nov 26, 2024 at 06:57:19PM -0800, Yuanchu Xie wrote: > > > > > This patch series provides workingset reporting of user pages in > > > > > lruvecs, of which coldness can be tracked by accessed bits and fd > > > > > references. However, the concept of workingset applies generically to > > > > > all types of memory, which could be kernel slab caches, discardable > > > > > userspace caches (databases), or CXL.mem. Therefore, data sources might > > > > > come from slab shrinkers, device drivers, or the userspace. > > > > > Another interesting idea might be hugepage workingset, so that we can > > > > > measure the proportion of hugepages backing cold memory. However, with > > > > > architectures like arm, there may be too many hugepage sizes leading to > > > > > a combinatorial explosion when exporting stats to the userspace. > > > > > Nonetheless, the kernel should provide a set of workingset interfaces > > > > > that is generic enough to accommodate the various use cases, and extensible > > > > > to potential future use cases. > > > > > > > > Doesn't DAMON already provide this information? > > > > > > > > CCing SJ. > > > Thanks for the CC. DAMON was really good at visualizing the memory > > > access frequencies last time I tried it out! > > > > Thank you for this kind acknowledgement, Yuanchu! > > > > > For server use cases, > > > DAMON would benefit from integrations with cgroups. The key then would be a > > > standard interface for exporting a cgroup's working set to the user. > > > > I show two ways to make DAMON supports cgroups for now. First way is making > > another DAMON operations set implementation for cgroups. I shared a rough idea > > for this before, probably on kernel summit. But I haven't had a chance to > > prioritize this so far. Please let me know if you need more details. The > > second way is extending DAMOS filter to provide more detailed statistics per > > DAMON-region, and adding another DAMOS action that does nothing but only > > accounting the detailed statistics. Using the new DAMOS action, users will be > > able to know how much of specific DAMON-found regions are filtered out by the > > given filter. Because we have DAMOS filter type for cgroups, we can know how > > much of workingset (or, warm memory) belongs to specific groups. This can be > > applied to not only cgroups, but for any DAMOS filter types that exist (e.g., > > anonymous page, young page). > > > > I believe the second way is simpler to implement while providing information > > that sufficient for most possible use cases. I was anyway planning to do this. I implemented the feature for the second approach I mentioned above. The initial version of the feature has recently merged[1] into the mainline as a part of 6.14-rc1 MM pull request. DAMON user-space tool (damo) is also updated for baisc support of it. I forgot updating that on this thread, sorry. > For a container orchestrator like kubernetes, the node agents need to > be able to gather the working set stats at a per-job level. Some jobs > can create sub-hierarchies as well, so it's important that we have > hierarchical stats. This makes sense to me. And yes, I believe DAMOS filters for memcg could also be used for this use case, since we can install and use multiple DAMOS filters in combinations. The documentation of the feature is not that good and there are many rooms to improve. You might not be able to get what you want in a perfect way with the current implementation. But we will continue improving it, and I believe we can make it faster if efforts are gathered. Of course, I could be wrong, and whether to use it or not is up to each person :) Anyway, please feel free to ask me questions or any help about the feature if you want. > > Do you think it's a good idea to integrate DAMON to provide some > aggregate stats in a memory controller file? With the DAMOS cgroup > filter, there can be some kind of interface that a DAMOS action or the > damo tool could call into. I feel that would be a straightforward and > integrated way to support cgroups. DAMON basically exposes its internal information via DAMON sysfs, and DAMON user-space tool (damo) uses it. In this case, per-memcg working set could also be retrieved in the way (directly from DAMON sysfs or indirectly from damo). But, yes, I think we could make new and optimized ABIs for exposing the information to user-space in more efficient way depending on the use case, if needed. DAMON modules such as DAMON_RECLAIM and DAMON_LRU_SORT provides their own ABIs that simplified and optimized for their usages. [1] https://git.kernel.org/torvalds/c/626ffabe67c2 Thanks, SJ [...]
