On Thu, Apr 15, 2021, at 6:11 PM Nitesh Narayan Lal <nitesh@xxxxxxxxxx> wrote: > > > On 4/14/21 12:11 PM, Jesse Brandeburg wrote: > > Nitesh Narayan Lal wrote: > > > >>> The original issue as seen, was that if you rmmod/insmod a driver > >>> *without* irqbalance running, the default irq mask is -1, which means > >>> any CPU. The older kernels (this issue was patched in 2014) used to use > >>> that affinity mask, but the value programmed into all the interrupt > >>> registers "actual affinity" would end up delivering all interrupts to > >>> CPU0, > >> So does that mean the affinity mask for the IRQs was different wrt where > >> the IRQs were actually delivered? > >> Or, the affinity mask itself for the IRQs after rmmod, insmod was changed > >> to 0 instead of -1? > > The smp_affinity was 0xfff, and the kernel chooses which interrupt to > > place the interrupt on, among any of the bits set. > > <snip> > > > > Your description of the problem makes it obvious there is an issue. It > > appears as if cpumask_local_spread() is the wrong function to use here. > > If you have any suggestions please let me know. > > > > We had one other report of this problem as well (I'm not sure if it's > > the same as your report) > > https://lkml.org/lkml/2021/3/28/206 > > https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20210125/023120.html > > So to understand further what the problem was with the older kernel based on Jesse's description and whether it is still there I did some more digging. Following are some of the findings (kindly correct me if there is a gap in my understanding): Part-1: Why there was a problem with the older kernel? ------ With a kernel built on top of the tag v4.0.0 (with Jesse's patch reverted and irqbalance disabled), if we observe the/proc/irq for ixgbe device IRQs then there are two things to note: # No separate effective affinity (Since it has been introduced as a part of the 2017 IRQ re-work) $ ls /proc/irq/86/ affinity_hint node p2p1 smp_affinity smp_affinity_list spurious # Multiple CPUs are set in the smp_affinity_list and the first CPU is CPU0: $ proc/irq/60/p2p1-TxRx-0 0,2,4,6,8,10,12,14,16,18,20,22 $ /proc/irq/61/p2p1-TxRx-1 0,2,4,6,8,10,12,14,16,18,20,22 $ /proc/irq/62/p2p1-TxRx-2 0,2,4,6,8,10,12,14,16,18,20,22 ... Now, if we read the commit message from Thomas's patch that was part of this IRQ re-work: fdba46ff: x86/apic: Get rid of multi CPU affinity " .. 2) Experiments have shown that the benefit of multi CPU affinity is close to zero and in some tests even worse than setting the affinity to a single CPU. The reason for this is that the delivery targets the APIC with the lowest ID first and only if that APIC is busy (servicing an interrupt, i.e. ISR is not empty) it hands it over to the next APIC. In the conducted tests the vast majority of interrupts ends up on the APIC with the lowest ID anyway, so there is no natural spreading of the interrupts possible.” " I think this explains why even if we have multiple CPUs in the SMP affinity mask the interrupts may only land on CPU0. With Jesse's patch in the kernel initial affinity mask that included multiple CPUs is overwritten with a single CPU. This CPU was previously selected based on vector_index, later on, this has been replaced with a logic where the CPU was fetched from cpumask_local_spread. Hence, in this case, the interrupts will be spread across to different CPUs. # listing the IRQ smp_affinity_list on the v4.0.0 kernel with Jesse's patch $ /proc/irq/60/p2p1-TxRx-0 0 $ /proc/irq/61/p2p1-TxRx-1 1 $ /proc/irq/62/p2p1-TxRx-2 2 ... $ /proc/irq/65/p2p1-TxRx-5 5 $ /proc/irq/66/p2p1-TxRx-6 6 ... Part-2: Why this may not be a problem anymore? ------ With the latest kernel, if we check the effective_affinity_list for i40e IRQs without irqblance and with Jesse's patch reverted, it is already set to a single CPU that is not always 0. This CPU is retrieved based on vector allocation count i.e., we get a CPU that has the lowest vector allocation count. $ /proc/irq/100/i40e-eno1-TxRx-5 28 $ /proc/irq/101/i40e-eno1-TxRx-6 30 $ /proc/irq/102/i40e-eno1-TxRx-7 32 … $ /proc/irq/121/i40e-eno1-TxRx-18 16 $ /proc/irq/122/i40e-eno1-TxRx-19 18 .. @Jesse do you think the Part-1 findings explain the behavior that you have observed in the past? Also, let me know if there are any suggestions or experiments to try here. -- Thanks Nitesh
