On Mon, 2021-01-11 at 16:21 +0000, John Garry wrote: > Hi, > > I was looking at some IOMMU issue on a LSI RAID 3008 card, and > noticed that performance there is not what I get on other SAS HBAs - > it's lower. > > After some debugging and fiddling with sdev queue depth in mpt3sas > driver, I am finding that performance changes appreciably with sdev > queue depth: > > sdev qdepth fio number jobs* 1 10 20 > 16 1590 1654 1660 > 32 1545 1646 1654 > 64 1436 1085 1070 > 254 (default) 1436 1070 1050 > > fio queue depth is 40, and I'm using 12x SAS SSDs. > > I got comparable disparity in results for fio queue depth = 128 and > num jobs = 1: > > sdev qdepth fio number jobs* 1 > 16 1640 > 32 1618 > 64 1577 > 254 (default) 1437 > > IO sched = none. > > That driver also sets queue depth tracking = 1, but never seems to > kick in. > > So it seems to me that the block layer is merging more bios per > request, as averge sg count per request goes up from 1 - > upto 6 or > more. As I see, when queue depth lowers the only thing that is really > changing is that we fail more often in getting the budget in > scsi_mq_get_budget()->scsi_dev_queue_ready(). > > So initial sdev queue depth comes from cmd_per_lun by default or > manually setting in the driver via scsi_change_queue_depth(). It > seems to me that some drivers are not setting this optimally, as > above. > > Thoughts on guidance for setting sdev queue depth? Could blk-mq > changed this behavior? In general, for spinning rust, you want the minimum queue depth possible for keeping the device active because merging is a very important performance enhancement and once the drive is fully occupied simply sending more tags won't improve latency. We used to recommend a depth of about 4 for this reason. A co-operative device can help you find the optimal by returning QUEUE_FULL when it's fully occupied so we have a mechanism to track the queue full returns and change the depth interactively. For high iops devices, these considerations went out of the window and it's generally assumed (without varying evidence) the more tags the better. SSDs have a peculiar lifetime problem in that when they get erase block starved they start behaving more like spinning rust in that they reach a processing limit but only for writes, so lowering the write queue depth (which we don't even have a knob for) might be a good solution. Trying to track the erase block problem has been a constant bugbear. I'm assuming you're using spinning rust in the above, so it sounds like the firmware in the card might be eating the queue full returns. I could see this happening in RAID mode, but it shouldn't happen in jbod mode. James
