On Thu, Jun 22, 2023 at 08:35:49AM -0600, Keith Busch wrote: > On Thu, Jun 22, 2023 at 09:51:12PM +0800, Ming Lei wrote: > > On Wed, Jun 21, 2023 at 09:48:49AM -0600, Keith Busch wrote: > > > The point was to contain requests from entering while the hctx's are > > > being reconfigured. If you're going to pair up the freezes as you've > > > suggested, we might as well just not call freeze at all. > > > > blk_mq_update_nr_hw_queues() requires queue to be frozen. > > It's too late at that point. Let's work through a real example. You'll > need a system that has more CPU's than your nvme has IO queues. > > Boot without any special nvme parameters. Every possible nvme IO queue > will be assigned "default" hctx type. Now start IO to every queue, then > run: > > # echo 8 > /sys/modules/nvme/parameters/poll_queues && echo 1 > /sys/class/nvme/nvme0/reset_controller > > Today, we freeze prior to tearing down the "default" IO queues, so > there's nothing entered into them while the driver reconfigures the > queues. nvme_start_freeze() just prevents new IO from being queued, and old ones may still be entering block layer queue, and what matters here is actually quiesce, which prevents new IO from being queued to driver/hardware. > > What you're suggesting will allow IO to queue up in a queisced "default" > queue, which will become "polled" without an interrupt hanlder on the > other side of the reset. The application doesn't know that, so the IO > you're allowing to queue up will time out. time out only happens after the request is queued to driver/hardware, or after blk_mq_start_request() is called in nvme_queue_rq(), but quiesce actually prevents new IOs from being dispatched to driver or be queued via .queue_rq(), meantime old requests have been canceled, so no any request can be timed out. Thanks, Ming
