On Thu, May 21, 2020 at 12:14:18AM +0200, Thomas Gleixner wrote: > Jens Axboe <axboe@xxxxxxxxx> writes: > > > On 5/20/20 1:41 PM, Thomas Gleixner wrote: > >> Jens Axboe <axboe@xxxxxxxxx> writes: > >>> On 5/20/20 8:45 AM, Jens Axboe wrote: > >>>> It just uses kthread_create_on_cpu(), nothing home grown. Pretty sure > >>>> they just break affinity if that CPU goes offline. > >>> > >>> Just checked, and it works fine for me. If I create an SQPOLL ring with > >>> SQ_AFF set and bound to CPU 3, if CPU 3 goes offline, then the kthread > >>> just appears unbound but runs just fine. When CPU 3 comes online again, > >>> the mask appears correct. > >> > >> When exactly during the unplug operation is it unbound? > > > > When the CPU has been fully offlined. I check the affinity mask, it > > reports 0. But it's still being scheduled, and it's processing work. > > Here's an example, PID 420 is the thread in question: > > > > [root@archlinux cpu3]# taskset -p 420 > > pid 420's current affinity mask: 8 > > [root@archlinux cpu3]# echo 0 > online > > [root@archlinux cpu3]# taskset -p 420 > > pid 420's current affinity mask: 0 > > [root@archlinux cpu3]# echo 1 > online > > [root@archlinux cpu3]# taskset -p 420 > > pid 420's current affinity mask: 8 > > > > So as far as I can tell, it's working fine for me with the goals > > I have for that kthread. > > Works for me is not really useful information and does not answer my > question: > > >> When exactly during the unplug operation is it unbound? > > The problem Ming and Christoph are trying to solve requires that the > thread is migrated _before_ the hardware queue is shut down and > drained. That's why I asked for the exact point where this happens. > > When the CPU is finally offlined, i.e. the CPU cleared the online bit in > the online mask is definitely too late simply because it still runs on > that outgoing CPU _after_ the hardware queue is shut down and drained. IMO, the patch in Christoph's blk-mq-hotplug.2 still works for percpu kthread. It is just not optimal in the retrying, but it should be fine. When the percpu kthread is scheduled on the CPU to be offlined: - if the kthread doesn't observe the INACTIVE flag, the allocated request will be drained. - otherwise, the kthread just retries and retries to allocate & release, and sooner or later, its time slice is consumed, and migrated out, and the cpu hotplug handler will get chance to run and move on, then the cpu is shutdown. - After the cpu is shutdown, the percpu kthread becomes unbound, and the allocation from new online cpu will succeed. Thanks, Ming
