On Fri, Feb 14, 2020 at 05:04:25PM +0100, Hannes Reinecke wrote: > On 2/14/20 3:40 PM, Keith Busch wrote: > > On Fri, Feb 14, 2020 at 08:32:57AM +0100, Hannes Reinecke wrote: > > > On 2/13/20 5:17 AM, Martin K. Petersen wrote: > > > > People often artificially lower the queue depth to avoid timeouts. The > > > > default timeout is 30 seconds from an I/O is queued. However, many > > > > enterprise applications set the timeout to 3-5 seconds. Which means that > > > > with deep queues you'll quickly start seeing timeouts if a drive > > > > temporarily is having issues keeping up (media errors, excessive spare > > > > track seeks, etc.). > > > > > > > > Well-behaved devices will return QF/TSF if they have transient resource > > > > starvation or exceed internal QoS limits. QF will cause the SCSI stack > > > > to reduce the number of I/Os in flight. This allows the drive to recover > > > > from its congested state and reduces the potential of application and > > > > filesystem timeouts. > > > > > > > This may even be a chance to revisit QoS / queue busy handling. > > > NVMe has this SQ head pointer mechanism which was supposed to handle > > > this kind of situations, but to my knowledge no-one has been > > > implementing it. > > > Might be worthwhile revisiting it; guess NVMe HDDs would profit from that. > > > > We don't need that because we don't allocate enough tags to potentially > > wrap the tail past the head. If you can allocate a tag, the queue is not > > full. And convesely, no tag == queue full. > > > It's not a problem on our side. > It's a problem on the target/controller side. > The target/controller might have a need to throttle I/O (due to QoS settings > or competing resources from other hosts), but currently no means of > signalling that to the host. > Which, incidentally, is the underlying reason for the DNR handling > discussion we had; NetApp tried to model QoS by sending "Namespace not > ready" without the DNR bit set, which of course is a totally different > use-case as the typical 'Namespace not ready' response we get (with the DNR > bit set) when a namespace was unmapped. > > And that is where SQ head pointer updates comes in; it would allow the > controller to signal back to the host that it should hold off sending I/O > for a bit. > So this could / might be used for NVMe HDDs, too, which also might have a > need to signal back to the host that I/Os should be throttled... Okay, I see. I think this needs a new nvme AER notice as Martin suggested. The desired host behavior is simiilar to what we do with a "firmware activation notice" where we temporarily quiesce new requests and reset IO timeouts for previously dispatched requests. Perhaps tie this to the CSTS.PP register as well.
