On 3/8/23 7:20?AM, Waiman Long wrote: > On 3/8/23 06:42, Daniel Dao wrote: >> Hi all, >> >> We encountered EINVAL when enabling cpuset in cgroupv2 when io_uring >> worker threads are running. Here are the steps to reproduce the failure >> on kernel 6.1.14: >> >> 1. Remove cpuset from subtree_control >> >> > for d in $(find /sys/fs/cgroup/ -maxdepth 1 -type d); do echo >> '-cpuset' | sudo tee -a $d/cgroup.subtree_control; done >> > cat /sys/fs/cgroup/cgroup.subtree_control >> cpu io memory pids >> >> 2. Run any applications that utilize the uring worker thread pool. I used >> https://github.com/cloudflare/cloudflare-blog/tree/master/2022-02-io_uring-worker-pool >> >> > cargo run -- -a -w 2 -t 2 >> >> 3. Enabling cpuset will return EINVAL >> >> > echo '+cpuset' | sudo tee -a /sys/fs/cgroup/cgroup.subtree_control >> +cpuset >> tee: /sys/fs/cgroup/cgroup.subtree_control: Invalid argument >> >> We traced this down to task_can_attach that will return EINVAL when it >> encounters >> kthreads with PF_NO_SETAFFINITY, which io_uring worker threads have. >> >> This seems like an unexpected interaction when enabling cpuset for the subtrees >> that contain kthreads. We are currently considering a workaround to try to >> enable cpuset in root subtree_control before any io_uring applications >> can start, >> hence failure to enable cpuset is localized to only cgroup with >> io_uring kthreads. >> But this is cumbersome. >> >> Any suggestions would be very much appreciated. > > Anytime you echo "+cpuset" to cgroup.subtree_control to enable cpuset, > the tasks within the child cgroups will do an implicit move from the > parent cpuset to the child cpusets. However, that move will fail if > any task has the PF_NO_SETAFFINITY flag set due to task_can_attach() > function which checks for this. One possible solution is for the > cpuset to ignore tasks with PF_NO_SETAFFINITY set for implicit move. > IOW, allowing the implicit move without touching it, but not explicit > one using cgroup.procs. I was pondering this too as I was typing my reply, but at least for io-wq, this report isn't the first to be puzzled or broken by the fact that task threads might have PF_NO_SETAFFINITY set. So while it might be worthwhile to for cpuset to ignore PF_NO_SETAFFINITY as a separate fix, I think it's better to fix io-wq in general. Not sure we have other cases where it's even possible to have PF_NO_SETAFFINITY set on userspace threads? -- Jens Axboe
