On 02/01/2017 08:07 AM, Kashyap Desai wrote:
-----Original Message-----
From: Hannes Reinecke [mailto:hare@xxxxxxx]
Sent: Wednesday, February 01, 2017 12:21 PM
To: Kashyap Desai; Christoph Hellwig
Cc: Martin K. Petersen; James Bottomley; linux-scsi@xxxxxxxxxxxxxxx;
Sathya
Prakash Veerichetty; PDL-MPT-FUSIONLINUX; Sreekanth Reddy
Subject: Re: [PATCH 00/10] mpt3sas: full mq support
On 01/31/2017 06:54 PM, Kashyap Desai wrote:
-----Original Message-----
From: Hannes Reinecke [mailto:hare@xxxxxxx]
Sent: Tuesday, January 31, 2017 4:47 PM
To: Christoph Hellwig
Cc: Martin K. Petersen; James Bottomley; linux-scsi@xxxxxxxxxxxxxxx;
Sathya
Prakash; Kashyap Desai; mpt-fusionlinux.pdl@xxxxxxxxxxxx
Subject: Re: [PATCH 00/10] mpt3sas: full mq support
On 01/31/2017 11:02 AM, Christoph Hellwig wrote:
On Tue, Jan 31, 2017 at 10:25:50AM +0100, Hannes Reinecke wrote:
Hi all,
this is a patchset to enable full multiqueue support for the
mpt3sas
driver.
While the HBA only has a single mailbox register for submitting
commands, it does have individual receive queues per MSI-X
interrupt and as such does benefit from converting it to full
multiqueue
support.
Explanation and numbers on why this would be beneficial, please.
We should not need multiple submissions queues for a single register
to benefit from multiple completion queues.
Well, the actual throughput very strongly depends on the blk-mq-sched
patches from Jens.
As this is barely finished I didn't post any numbers yet.
However:
With multiqueue support:
4k seq read : io=60573MB, bw=1009.2MB/s, iops=258353, runt=
60021msec
With scsi-mq on 1 queue:
4k seq read : io=17369MB, bw=296291KB/s, iops=74072, runt= 60028msec
So yes, there _is_ a benefit.
(Which is actually quite cool, as these tests were done on a SAS3
HBA,
so
we're getting close to the theoretical maximum of 1.2GB/s).
(Unlike the single-queue case :-)
Hannes -
Can you share detail about setup ? How many drives do you have and how
is connection (enclosure -> drives. ??) ?
To me it looks like current mpt3sas driver might be taking more hit in
spinlock operation (penalty on NUMA arch is more compare to single
core
server) unlike we have in megaraid_sas driver use of shared blk tag.
The tests were done with a single LSI SAS3008 connected to a NetApp E-
series (2660), using 4 LUNs under MD-RAID0.
Megaraid_sas is even worse here; due to the odd nature of the 'fusion'
implementation we're ending up having _two_ sets of tags, making it really
hard to use scsi-mq here.
Current megaraid_sas as single submission queue exposed to the blk-mq will
not encounter similar performance issue.
We may not see significant improvement of performance if we attempt the same
for megaraid_sas driver.
We had similar discussion for megaraid_sas and hpsa.
http://www.spinics.net/lists/linux-scsi/msg101838.html
I am seeing this patch series is similar attempt for mpt3sas..Am I missing
anything ?
No, you don't. That is precisely the case.
The different here is that mpt3sas is actually exposing hardware
capabilities, whereas with megaraid_sas (and hpsa) we're limited by the
hardware implementation to a single completion queue shared between HBA
and OS.
With mpt3sas we're having per-interrupt completion queues (well, for
newer firmware :-) so we can take advantage of scsi-mq.
(And if someone had done a _proper_ design of the megaraid_sas_fusion
thing by exposing several submission and completion queues for
megaraid_sas itself instead of bolting the existing megaraid_sas single
queue approach ontop of the mpt3sas multiqueue design we could have done
the same thing there ... sigh)
Megaraid_sas driver just do indexing from blk_tag and fire IO quick enough
unlike mpt3sas where we have lock contention @driver level as bottleneck.
(Not that I didn't try; but lacking a proper backend it's really hard to
evaluate
the benefit of those ... spinning HDDs simply don't cut it here)
I mean " [PATCH 08/10] mpt3sas: lockless command submission for scsi-
mq"
patch is improving performance removing spinlock overhead and
attempting to get request using blk_tags.
Are you seeing performance improvement if you hard code nr_hw_queues
= 1 in below code changes part of "[PATCH 10/10] mpt3sas: scsi-mq
interrupt steering"
No. The numbers posted above are generated with exactly that patch; the
first line is running with nr_hw_queues=32 and the second line with
nr_hw_queues=1.
Thanks Hannes. That clarifies. Can you share <fio> script you have used ?
If my understanding correct, you will see theoretical maximum of 1.2GBp/s
if you restrict your work load to single numa node. This is just for
understanding if <mpt3sas> driver spinlocks are adding overhead. We have
seen such overhead on multi-socket server and it is reasonable to reduce
lock in mpt3sas driver, but only concern is exposing HBA for multiple
submission queue to blk-mq is really not required and trying to figure out
if we have any side effect of doing that.
Well, the HBA has per-MSIx completion queues, so I don't see any issues
with exposing them.
blk-mq is designed to handle per-CPU queues, so exposing all hardware
queues will be beneficial especially in a low-latency context; and as
the experiments show, even when connected to an external storage there
is a benefit to be had.
But exposing all queues might even reduce or even resolve your FW Fault
status 0x2100 state; with that patch you now have each queue pulling
request off the completion queue and updating the reply post host index
in parallel, making the situation far more unlikely.
Cheers,
Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@xxxxxxx +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
