On Sat, Aug 24, 2019 at 12:27:18AM +0000, Long Li wrote: > >>>Subject: Re: [PATCH 3/3] nvme: complete request in work queue on CPU > >>>with flooded interrupts > >>> > >>>On Tue, Aug 20, 2019 at 10:33:38AM -0700, Sagi Grimberg wrote: > >>>> > >>>> > From: Long Li <longli@xxxxxxxxxxxxx> > >>>> > > >>>> > When a NVMe hardware queue is mapped to several CPU queues, it is > >>>> > possible that the CPU this hardware queue is bound to is flooded by > >>>> > returning I/O for other CPUs. > >>>> > > >>>> > For example, consider the following scenario: > >>>> > 1. CPU 0, 1, 2 and 3 share the same hardware queue 2. the hardware > >>>> > queue interrupts CPU 0 for I/O response 3. processes from CPU 1, 2 > >>>> > and 3 keep sending I/Os > >>>> > > >>>> > CPU 0 may be flooded with interrupts from NVMe device that are I/O > >>>> > responses for CPU 1, 2 and 3. Under heavy I/O load, it is possible > >>>> > that CPU 0 spends all the time serving NVMe and other system > >>>> > interrupts, but doesn't have a chance to run in process context. > >>>> > > >>>> > To fix this, CPU 0 can schedule a work to complete the I/O request > >>>> > when it detects the scheduler is not making progress. This serves > >>>multiple purposes: > >>>> > > >>>> > 1. This CPU has to be scheduled to complete the request. The other > >>>> > CPUs can't issue more I/Os until some previous I/Os are completed. > >>>> > This helps this CPU get out of NVMe interrupts. > >>>> > > >>>> > 2. This acts a throttling mechanisum for NVMe devices, in that it > >>>> > can not starve a CPU while servicing I/Os from other CPUs. > >>>> > > >>>> > 3. This CPU can make progress on RCU and other work items on its > >>>queue. > >>>> > >>>> The problem is indeed real, but this is the wrong approach in my mind. > >>>> > >>>> We already have irqpoll which takes care proper budgeting polling > >>>> cycles and not hogging the cpu. > >>> > >>>The issue isn't unique to NVMe, and can be any fast devices which > >>>interrupts CPU too frequently, meantime the interrupt/softirq handler may > >>>take a bit much time, then CPU is easy to be lockup by the interrupt/sofirq > >>>handler, especially in case that multiple submission CPUs vs. single > >>>completion CPU. > >>> > >>>Some SCSI devices has the same problem too. > >>> > >>>Could we consider to add one generic mechanism to cover this kind of > >>>problem? > >>> > >>>One approach I thought of is to allocate one backup thread for handling such > >>>interrupt, which can be marked as IRQF_BACKUP_THREAD by drivers. > >>> > >>>Inside do_IRQ(), irqtime is accounted, before calling action->handler(), > >>>check if this CPU has taken too long time for handling IRQ(interrupt or > >>>softirq) and see if this CPU could be lock up. If yes, wakeup the backup > > How do you know if this CPU is spending all the time in do_IRQ()? > > Is it something like: > If (IRQ_time /elapsed_time > a threshold value) > wake up the backup thread Yeah, the above could work in theory. Another approach I thought of is to monitor average irq gap time on each CPU. We could use EWMA(Exponential Weighted Moving Average) to do it simply, such as: curr_irq_gap(cpu) = current start time of do_IRQ() on 'cpu' - end time of last do_IRQ() on 'cpu' avg_irq_gap(cpu) = weight_prev * avg_irq_gap(cpu) + weight_curr * curr_irq_gap(cpu) note: weight_prev + weight_curr = 1 When avg_irq_gap(cpu) is close to one small enough threshold, we think irq flood is detected. 'weight_prev' could be chosen as one big enough value for avoiding short-time flood. Thanks, Ming
