On Mon, Jan 11, 2021 at 04:21:27PM +0000, John Garry wrote: > Hi, > > I was looking at some IOMMU issue on a LSI RAID 3008 card, and noticed that > performance there is not what I get on other SAS HBAs - it's lower. > > After some debugging and fiddling with sdev queue depth in mpt3sas driver, I > am finding that performance changes appreciably with sdev queue depth: > > sdev qdepth fio number jobs* 1 10 20 > 16 1590 1654 1660 > 32 1545 1646 1654 > 64 1436 1085 1070 > 254 (default) 1436 1070 1050 What does the performance number mean? IOPS or others? What is the fio io test? random IO or sequential IO? > > fio queue depth is 40, and I'm using 12x SAS SSDs. > > I got comparable disparity in results for fio queue depth = 128 and num jobs > = 1: > > sdev qdepth fio number jobs* 1 > 16 1640 > 32 1618 > 64 1577 > 254 (default) 1437 > > IO sched = none. > > That driver also sets queue depth tracking = 1, but never seems to kick in. > > So it seems to me that the block layer is merging more bios per request, as > averge sg count per request goes up from 1 - > upto 6 or more. As I see, > when queue depth lowers the only thing that is really changing is that we > fail more often in getting the budget in > scsi_mq_get_budget()->scsi_dev_queue_ready(). Right, the behavior basically doesn't change compared with block legacy io path. And that is why sdev->queue_depth is a bit important for HDD. > > So initial sdev queue depth comes from cmd_per_lun by default or manually > setting in the driver via scsi_change_queue_depth(). It seems to me that > some drivers are not setting this optimally, as above. > > Thoughts on guidance for setting sdev queue depth? Could blk-mq changed this > behavior? So far, the sdev queue depth is provided by SCSI layer, and blk-mq can queue one request only if budget is obtained via .get_budget(). Thanks, Ming
