On 10/26/2020 3:13 PM, Jakub Kicinski wrote: > On Mon, 26 Oct 2020 22:50:45 +0100 Thomas Gleixner wrote: >> On Mon, Oct 26 2020 at 14:11, Jacob Keller wrote: >>> On 10/26/2020 1:11 PM, Thomas Gleixner wrote: >>>> On Mon, Oct 26 2020 at 12:21, Jacob Keller wrote: >>>>> Are there drivers which use more than one interrupt per queue? I know >>>>> drivers have multiple management interrupts.. and I guess some drivers >>>>> do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to >>>>> have multiple queues for one interrupt .. I'm not sure how a single >>>>> queue with multiple interrupts would work though. >>>> >>>> For block there is always one interrupt per queue. Some Network drivers >>>> seem to have seperate RX and TX interrupts per queue. >>> That's true when thinking of Tx and Rx as a single queue. Another way to >>> think about it is "one rx queue" and "one tx queue" each with their own >>> interrupt... >>> >>> Even if there are devices which force there to be exactly queue pairs, >>> you could still think of them as separate entities? >> >> Interesting thought. >> >> But as Jakub explained networking queues are fundamentally different >> from block queues on the RX side. For block the request issued on queue >> X will raise the complete interrupt on queue X. >> >> For networking the TX side will raise the TX interrupt on the queue on >> which the packet was queued obviously or should I say hopefully. :) >> >> But incoming packets will be directed to some receive queue based on a >> hash or whatever crystallball logic the firmware decided to implement. >> >> Which makes this not really suitable for the managed interrupt and >> spreading approach which is used by block-mq. Hrm... >> >> But I still think that for curing that isolation stuff we want at least >> some information from the driver. Alternative solution would be to grant >> the allocation of interrupts and queues and have some sysfs knob to shut >> down queues at runtime. If that shutdown results in releasing the queue >> interrupt (via free_irq()) then the vector exhaustion problem goes away. >> >> Needs more thought and information (for network oblivious folks like >> me). > > One piece of information that may be useful is that even tho the RX > packets may be spread semi-randomly the user space can still control > which queues are included in the mechanism. There is an indirection > table in the HW which allows to weigh queues differently, or exclude > selected queues from the spreading. Other mechanisms exist to filter > flows onto specific queues. > > IOW just because a core has an queue/interrupt does not mean that > interrupt will ever fire, provided its excluded from RSS. > > Another piece is that by default we suggest drivers allocate 8 RX > queues, and online_cpus TX queues. The number of queues can be > independently controlled via ethtool -L. Drivers which can't support > separate queues will default to online_cpus queue pairs, and let > ethtool -L only set the "combined" parameter. > I know the Intel drivers usually have defaulted to trying to maintain queue pairs. I do not believe this is technically a HW restriction, but it is heavily built into the way the drivers work today. > There are drivers which always allocate online_cpus interrupts, > and then some of them will go unused if #qs < #cpus. > > Right. > My unpopular opinion is that for networking devices all the heuristics > we may come up with are going to be a dead end. We need an explicit API > to allow users placing queues on cores, and use managed IRQs for data > queues. (I'm assuming that managed IRQs will let us reliably map a MSI-X > vector to a core :)) > I don't think it is that unpopular... This is the direction I'd like to see us go as well.
