On 1/11/21 5:21 PM, John Garry wrote:
Hi,
I was looking at some IOMMU issue on a LSI RAID 3008 card, and noticed
that performance there is not what I get on other SAS HBAs - it's lower.
After some debugging and fiddling with sdev queue depth in mpt3sas
driver, I am finding that performance changes appreciably with sdev
queue depth:
sdev qdepth fio number jobs* 1 10 20
16 1590 1654 1660
32 1545 1646 1654
64 1436 1085 1070
254 (default) 1436 1070 1050
fio queue depth is 40, and I'm using 12x SAS SSDs.
I got comparable disparity in results for fio queue depth = 128 and num
jobs = 1:
sdev qdepth fio number jobs* 1
16 1640
32 1618
64 1577
254 (default) 1437
IO sched = none.
That driver also sets queue depth tracking = 1, but never seems to kick in.
So it seems to me that the block layer is merging more bios per request,
as averge sg count per request goes up from 1 - > upto 6 or more. As I
see, when queue depth lowers the only thing that is really changing is
that we fail more often in getting the budget in
scsi_mq_get_budget()->scsi_dev_queue_ready().
So initial sdev queue depth comes from cmd_per_lun by default or
manually setting in the driver via scsi_change_queue_depth(). It seems
to me that some drivers are not setting this optimally, as above.
Thoughts on guidance for setting sdev queue depth? Could blk-mq changed
this behavior?
First of all: are these 'real' SAS SSDs?
The peak at 32 seems very ATA-ish, and I wouldn't put it past the LSI
folks to optimize for that case :-)
Can you get a more detailed picture by changing the queue depth more
finegrained?
(Will get you nicer graphs to boot :-)
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@xxxxxxx +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
