在 2024/3/27 01:26, Tejun Heo 写道:
Hello, On Mon, Mar 25, 2024 at 10:46:09PM +0800, Chuyi Zhou wrote:In our production environment, we have observed several cases of hung tasks blocked on the cgroup_mutex. The underlying cause is that when user modify the cpuset.mems, memory migration operations are performed in the work_queue. However, the duration of these operations depends on the memory size of workloads and can consume a significant amount of time. In the __cgroup_procs_write operation, there is a flush_workqueue operation that waits for the migration to complete while holding the cgroup_mutex. As a result, most cgroup-related operations have the potential to experience blocking. We have noticed the commit "cgroup/cpuset: Enable memory migration for cpuset v2"[1]. This commit enforces memory migration when modifying the cpuset. Furthermore, in cgroup v2, there is no option available for users to disable CS_MEMORY_MIGRATE. In our scenario, we do need to perform memory migration when cpuset.mems changes, while ensuring that other tasks are not blocked on cgroup_mutex for an extended period of time. One feasible approach is to revert the commit "cgroup/cpuset: Enable memory migration for cpuset v2"[1]. This way, modifying cpuset.mems will not trigger memory migration, and we can manually perform memory migration using migrate_pages()/move_pages() syscalls. Another solution is to use a lazy approach for memory migration[2]. In this way we only walk through all the pages and sets pages to protnone, and numa faults triggered by later touch will handle the movement. That would significantly reduce the time spent in cpuset_migrate_mm_workfn. But MPOL_MF_LAZY was disabled by commit 2cafb582173f ("mempolicy: remove confusing MPOL_MF_LAZY dead code")One approach we can take is pushing the cpuset_migrate_mm_wq flushing to task_work so that it happens after cpuset mutex is dropped. That way we maintain the operation synchronicity for the issuer while avoiding bothering anyone else. Can you see whether the following patch fixes the issue for you? Thanks.
Thanks for the reply! I think it would help to fix this issue. BTW, will you merge this patch to the mainline?
