Thanks, Neil. Let me see what I can find out.
Regards,
Mohsin
On Mon, Nov 23, 2015 at 8:55 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
> On Fri, Nov 20, 2015 at 04:52:31PM -0500, Mohsin Zaidi wrote:
>> I stopped irqbalance and set the affinities of all the interface IRQs
>> to 18 and only 169 of them get set correctly to 18. The rest have the
>> same value they had before (in my test 19,55).
>> Regards,
>> Mohsin
>>
> Ok, thats starting to make more sense. That indicates the issue is occuring in
> the kernel, not irqbalance. Whats most likely happening is either the code that
> parses the cpu mask buffer has a bug that can't handle a mask that big (unlikley
> given this works on other simmilarly sized hardware), or something about the
> irq controllers irq_set_affinity method somehow can't handle the mask that
> you're providing.
>
> The next step would be to enable dynamic debug in the kernel for all files in
> arch/x86/kernel and below, and crank your console printk level up to DEBUG, to
> see if any errors are reported when setting affinities on some of those
> interrupts.
>
> Neil
>
>>
>> On Fri, Nov 20, 2015 at 4:23 PM, Neil Horman <nhorman at tuxdriver.com> wrote:
>> > On Fri, Nov 20, 2015 at 01:45:37PM -0500, Mohsin Zaidi wrote:
>> >> Some more observations.
>> >>
>> >> When I said yesterday that changing the unbanned CPUs to 19/55 or
>> >> 18/54 worked correctly for all IRQs, I failed to notice that of the
>> >> 256 IRQs for the interfaces, 3 would never have their affinities get
>> >> updated correctly.
>> >>
>> >> For example, with the banning mask set to "ff,ff7fffff,fff7ffff", the
>> >> smp_affinity_list values for the last 10 IRQs are as follows:
>> >>
>> >> 19
>> >> 55
>> >> 26
>> >> 55
>> >> 24
>> >> 55
>> >> 19
>> >> 19
>> >> 19
>> >> 22
>> >>
>> >> 3 of these are set to whatever was set for them last (my last test was
>> >> to unban all CPUs). I see this pattern repeated every time.
>> >>
>> >> I changed the test to unban 18-19,54-55 at the same time, and this
>> >> problem went away. When I unbanned just 19/55 and reduced the number
>> >> of queues per interface by one, the problem also went away.
>> >>
>> >> It's as if 2 CPUs can't be successfully assigned 256 IRQs. This also
>> >> holds true if the CPUs are not siblings (e.g. 19/54).
>> >>
>> > I wonder if this is a hardware limitation (i.e. if you're hitting the upper
>> > limit of the elligible cpu set in an MSI write or some such).
>> >
>> > If you manually set all irqs to a single cpu, what happens?
>> >
>> > Neil
>> >
>> >> So there are two dimensions to the problem. One is choosing CPUs just
>> >> on NUMA node 0 doesn't work, and the other is that assigning 256 IRQs
>> >> to 2 CPUs on NUMA node 1 doesn't work.
>> >> Regards,
>> >> Mohsin
>> >>
>> >>
>> >> On Fri, Nov 20, 2015 at 9:45 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
>> >> > On Thu, Nov 19, 2015 at 01:32:58PM -0500, Mohsin Zaidi wrote:
>> >> >> Thanks, Neil. I'll have the results for you shortly.
>> >> >>
>> >> >> I wanted to point out that each of the 4 interfaces on the server have
>> >> >> 64 queues, so there are a total of 256 queues. Also, the banning is
>> >> >> attempting to direct interrupts to just two processors (#1 and #37) on
>> >> >> the same NUMA node, which is also not the same as the NUMA node that
>> >> >> "owns" the interface I am looking at (eth03).
>> >> >>
>> >> >> Does any of this matter?
>> >> > It really shouldn't, but given that I'm at a loss to explain the behavior yet,
>> >> > anything is on the table.
>> >> > Neil
>> >> >
>> >> >> Regards,
>> >> >> Mohsin
>> >> >>
>> >> >>
>> >> >> On Thu, Nov 19, 2015 at 9:58 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
>> >> >> > On Wed, Nov 18, 2015 at 10:42:41AM -0500, Mohsin Zaidi wrote:
>> >> >> >> I'm using the irqbalance daemon with the following config file. The
>> >> >> >> only thing I've changed is the banned CPUs list, and I've banned all
>> >> >> >> but CPUs #1 and #37. Interrupts *never* go to #1, and go to #18 and
>> >> >> >> #37, even though #18 has also been banned.
>> >> >> >>
>> >> >> >> # irqbalance is a daemon process that distributes interrupts across
>> >> >> >> # CPUS on SMP systems. The default is to rebalance once every 10
>> >> >> >> # seconds. This is the environment file that is specified to systemd via the
>> >> >> >> # EnvironmentFile key in the service unit file (or via whatever method the init
>> >> >> >> # system you're using has.
>> >> >> >> #
>> >> >> >> # ONESHOT=yes
>> >> >> >> # after starting, wait for a minute, then look at the interrupt
>> >> >> >> # load and balance it once; after balancing exit and do not change
>> >> >> >> # it again.
>> >> >> >> #IRQBALANCE_ONESHOT=
>> >> >> >>
>> >> >> >> #
>> >> >> >> # IRQBALANCE_BANNED_CPUS
>> >> >> >> # 64 bit bitmask which allows you to indicate which cpu's should
>> >> >> >> # be skipped when reblancing irqs. Cpu numbers which have their
>> >> >> >> # corresponding bits set to one in this mask will not have any
>> >> >> >> # irq's assigned to them on rebalance
>> >> >> >> #
>> >> >> >> #IRQBALANCE_BANNED_CPUS=
>> >> >> >> IRQBALANCE_BANNED_CPUS=000000ff,ffffffdf,fffffffd
>> >> >> >>
>> >> >> >> #
>> >> >> >> # IRQBALANCE_ARGS
>> >> >> >> # append any args here to the irqbalance daemon as documented in the man page
>> >> >> >> #
>> >> >> >> #IRQBALANCE_ARGS=
>> >> >> >> Regards,
>> >> >> >> Mohsin
>> >> >> >>
>> >> >> >>
>> >> >> >> On Wed, Nov 18, 2015 at 10:28 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
>> >> >> >> > On Wed, Nov 18, 2015 at 10:04:56AM -0500, Mohsin Zaidi wrote:
>> >> >> >> >> Sorry about that, Neil.
>> >> >> >> >>
>> >> >> >> >> I haven't specified any hint policy in IRQBALANCE_ARGS (for the daemon).
>> >> >> >> >> Regards,
>> >> >> >> >> Mohsin
>> >> >> >> >>
>> >> >> >> > Ok, well, I'm at a bit of a loss. irqbalance, based on your output from the
>> >> >> >> > debug log, is working properly, presuming you actually listed cpus 18 and 37 as
>> >> >> >> > your only unbanned one, which you indicate is the opposite of what you've
>> >> >> >> > configured.
>> >> >> >> >
>> >> >> >> > Can you please send me the command line you use to start irqbalance?
>> >> >> >> >
>> >> >> >> > Neil
>> >> >> >> >
>> >> >> >> >>
>> >> >> >> >> On Wed, Nov 18, 2015 at 6:36 AM, Neil Horman <nhorman at tuxdriver.com> wrote:
>> >> >> >> >> > On Fri, Nov 13, 2015 at 04:39:08PM -0500, Neil Horman wrote:
>> >> >> >> >> >> On Fri, Nov 13, 2015 at 01:39:20PM -0500, Mohsin Zaidi wrote:
>> >> >> >> >> >> > Thanks for your reply, Neil.
>> >> >> >> >> >> >
>> >> >> >> >> >> > Yes, when I manually set the irq affinity to avoid #18, it works.
>> >> >> >> >> >> >
>> >> >> >> >> >> > I just downloaded and applied the latest irqbalance code, but it's
>> >> >> >> >> >> > showing the same behavior.
>> >> >> >> >> >> >
>> >> >> >> >> >> What hint policy are you using?
>> >> >> >> >> >>
>> >> >> >> >> >> Neil
>> >> >> >> >> >>
>> >> >> >> >> > Ping, any response regarding hint policy?
>> >> >> >> >> >
>> >> >> >> >> > Neil
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> > I'm at something of a loss here. I can see no reason why this would fail on
>> >> >> > only one system. In an effort to get additional data, please apply this patch,
>> >> >> > run irqbalance in debug mode and post the output please.
>> >> >> >
>> >> >> > Thanks!
>> >> >> > Neil
>> >> >> >
>> >> >> >
>> >> >> > diff --git a/activate.c b/activate.c
>> >> >> > index c8453d5..d92e770 100644
>> >> >> > --- a/activate.c
>> >> >> > +++ b/activate.c
>> >> >> > @@ -113,6 +113,7 @@ static void activate_mapping(struct irq_info *info, void *data __attribute__((un
>> >> >> > return;
>> >> >> >
>> >> >> > cpumask_scnprintf(buf, PATH_MAX, applied_mask);
>> >> >> > + printf("Applying mask for irq %d: 5s\n", info->irq, buf);
>> >> >> > fprintf(file, "%s", buf);
>> >> >> > fclose(file);
>> >> >> > info->moved = 0; /*migration is done*/
>> >> >>
>> >>
>>
