On Wed, Mar 8, 2023 at 2:27 PM Jens Axboe <axboe@xxxxxxxxx> wrote: > > Every now and then reports come in that are puzzled on why changing > affinity on the io-wq workers fails with EINVAL. This happens because they > set PF_NO_SETAFFINITY as part of their creation, as io-wq organizes > workers into groups based on what CPU they are running on. > > However, this is purely an optimization and not a functional requirement. > We can allow setting affinity, and just lazily update our worker to wqe > mappings. If a given io-wq thread times out, it normally exits if there's > no more work to do. The exception is if it's the last worker available. > For the timeout case, check the affinity of the worker against group mask > and exit even if it's the last worker. New workers should be created with > the right mask and in the right location. The patch resolved the bug around enabling cpuset for subtree_control for me. However, it also doesn't prevent user from setting cpuset value that is incompatible with iou threads. For example, on a 2-numa 4-cpu node, new iou-wrks are bound to 2-3 while we can set cpuset.cpus to 1-2 successfully. The end result is a mix of cpu distribution such as: pid 533's current affinity list: 1,2 # process pid 720's current affinity list: 1,2 # iou-wrk-533 pid 5236's current affinity list: 2,3 # iou-wrk-533, running outside of cpuset IMO this violated the principle of cpuset and can be confusing for end users. I think I prefer Waiman's suggestion of allowing an implicit move to cpuset when enabling cpuset with subtree_control but not explicit moves such as when setting cpuset.cpus or writing the pids into cgroup.procs. It's easier to reason about and make the failure mode more explicit. What do you think ? Cheers, Daniel.
