Alexey Klimov <aklimov@xxxxxxxxxx> writes:
> When a CPU offlined and onlined via device_offline() and device_online()
> the userspace gets uevent notification. If, after receiving "online" uevent,
> userspace executes sched_setaffinity() on some task trying to move it
> to a recently onlined CPU, then it often fails with -EINVAL. Userspace needs
> to wait around 5..30 ms before sched_setaffinity() will succeed for the recently
> onlined CPU after receiving uevent.
>
> If in_mask argument for sched_setaffinity() has only recently onlined CPU,
> it often fails with such flow:
>
> sched_setaffinity()
> cpuset_cpus_allowed()
> guarantee_online_cpus() <-- cs->effective_cpus mask does not
> contain recently onlined cpu
> cpumask_and() <-- final new_mask is empty
> __set_cpus_allowed_ptr()
> cpumask_any_and_distribute() <-- returns dest_cpu equal to nr_cpu_ids
> returns -EINVAL
>
> Cpusets used in guarantee_online_cpus() are updated using workqueue from
> cpuset_update_active_cpus() which in its turn is called from cpu hotplug callback
> sched_cpu_activate() hence it may not be observable by sched_setaffinity() if
> it is called immediately after uevent.
> Out of line uevent can be avoided if we will ensure that cpuset_hotplug_work
> has run to completion using cpuset_wait_for_hotplug() after onlining the
> cpu in cpu_up() and in cpuhp_smt_enable().
Nice writeup. I just have some nits, patch looks ok otherwise.
> @@ -1281,6 +1282,11 @@ static int cpu_up(unsigned int cpu, enum cpuhp_state target)
> err = _cpu_up(cpu, 0, target);
> out:
> cpu_maps_update_done();
> +
> + /* To avoid out of line uevent */
Not sure this will make sense out of context. Maybe,
/*
* Wait for cpuset updates to cpumasks to finish. Later on this path
* may generate uevents whose consumers rely on the updates.
*/
> @@ -2062,8 +2068,6 @@ static void cpuhp_offline_cpu_device(unsigned int cpu)
> struct device *dev = get_cpu_device(cpu);
>
> dev->offline = true;
> }
>
> @@ -2071,14 +2075,18 @@ static void cpuhp_online_cpu_device(unsigned int cpu)
> struct device *dev = get_cpu_device(cpu);
>
> dev->offline = false;
> }
You could get rid of these functions and just put the few remaining bits
in the callers. They each have only one.
