On 11/03/14 02:46, Elliott, Robert (Server Storage) wrote:
-----Original Message-----
From: Sagi Grimberg [mailto:sagig@xxxxxxxxxxxxxxxxxx]
Sent: Sunday, November 02, 2014 7:03 AM
To: Bart Van Assche; Christoph Hellwig
Cc: Jens Axboe; Sagi Grimberg; Sebastian Parschauer; Elliott, Robert
(Server Storage); Ming Lei; linux-scsi@xxxxxxxxxxxxxxx; linux-rdma
Subject: Re: [PATCH v2 12/12] IB/srp: Add multichannel support
...
IMHO, this is not iSER specific issue, it is easily indicated from the
code that a specific workload SRP will poll recv completion queue
forever in an interrupt context.
I encountered this issue on a virtual guest in a high workload (80+
sessions with heavy traffic on all) because qemu smp_affinity setting
was broken (might still be, didn't check that for a while). This caused
all completion vectors to fire interrupts to core 0 causing a high
events contention on a single event queue (causing lockup situations
and starvation of other CQs). Using more completion queues will enhance
this situation.
I think running multichannel code when all MSIX vectors affinity are
directed to a single CPU can invoke what I'm talking about.
That's not an SRP specific problem either. If you ask just one CPU to
service interrupts and block layer completions for submissions from lots
of other CPUs, it's bound to become overloaded.
Setting rq_affinity=2 helps quite a bit for the block layer completion
work. This patch proposed making that the default for blk-mq:
https://lkml.org/lkml/2014/9/9/931
For SRP interrupt processing, irqbalance recently changed its default
to ignore the affinity_hint; you now need to pass an option to honor
the hint, or provide a policy script to do so for selected irqs. For
multi-million IOPS workloads, irqbalance takes far too long to reroute
them based on activity; you're likely to overload a CPU with 100%
hardirq processing, creating self-detected stalls for the submitting
processes on that CPU and other problems. Sending interrupts back
to the submitting CPU provides self-throttling.
Hello Sagi,
To me it seems like with Rob's reply all questions about this patch
series have been answered. But I think Christoph is still waiting for a
Reviewed-by tag from you for patch 12/12.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
