On Wed, Jul 06, 2022 at 06:30:43PM +0300, Sagi Grimberg wrote: > > > > > update the subject to better describe the issue: > > > > > > > > So I tried this issue on one nvme/rdma environment, and it was also > > > > reproducible, here are the steps: > > > > > > > > # echo 0 >/sys/devices/system/cpu/cpu0/online > > > > # dmesg | tail -10 > > > > [ 781.577235] smpboot: CPU 0 is now offline > > > > # nvme connect -t rdma -a 172.31.45.202 -s 4420 -n testnqn > > > > Failed to write to /dev/nvme-fabrics: Invalid cross-device link > > > > no controller found: failed to write to nvme-fabrics device > > > > > > > > # dmesg > > > > [ 781.577235] smpboot: CPU 0 is now offline > > > > [ 799.471627] nvme nvme0: creating 39 I/O queues. > > > > [ 801.053782] nvme nvme0: mapped 39/0/0 default/read/poll queues. > > > > [ 801.064149] nvme nvme0: Connect command failed, error wo/DNR bit: -16402 > > > > [ 801.073059] nvme nvme0: failed to connect queue: 1 ret=-18 > > > > > > This is because of blk_mq_alloc_request_hctx() and was raised before. > > > > > > IIRC there was reluctance to make it allocate a request for an hctx even > > > if its associated mapped cpu is offline. > > > > > > The latest attempt was from Ming: > > > [PATCH V7 0/3] blk-mq: fix blk_mq_alloc_request_hctx > > > > > > Don't know where that went tho... > > > > The attempt relies on that the queue for connecting io queue uses > > non-admined irq, unfortunately that can't be true for all drivers, > > so that way can't go. > > The only consumer is nvme-fabrics, so others don't matter. > Maybe we need a different interface that allows this relaxation. > > > So far, I'd suggest to fix nvme_*_connect_io_queues() to ignore failed > > io queue, then the nvme host still can be setup with less io queues. > > What happens when the CPU comes back? Not sure we can simply ignore it. Anyway, it is a not good choice to fail the whole controller if only one queue can't be connected. I meant the queue can be kept as non-LIVE, and it should work since no any io can be issued to this queue when it is non-LIVE. Just wondering why we can't re-connect the io queue and set LIVE after any CPU in the this hctx->cpumask becomes online? blk-mq could add one pair of callbacks for driver for handing this queue change. thanks, Ming
