On 9/23/20 3:33 AM, Ming Lei wrote:
scsi requires one global atomic variable to track queue depth for each LUN/ request queue, meantime blk-mq tracks queue depth for each hctx. This SCSI's requirement can't be implemented in blk-mq easily, cause it is a bigger & harder problem to spread the device or request queue's depth among all hw queues. The current approach by using atomic variable can't scale well when there is lots of CPU cores and the disk is very fast and IO are submitted to this device concurrently. It has been observed that IOPS is affected a lot by tracking queue depth via sdev->device_busy in IO path. So replace the atomic variable sdev->device_busy with sbitmap for tracking scsi device queue depth. It is observed that IOPS is improved ~30% by this patchset in the following test: 1) test machine(32 logical CPU cores) Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 2 Model name: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz 2) setup scsi_debug: modprobe scsi_debug virtual_gb=128 max_luns=1 submit_queues=32 delay=0 max_queue=256 3) fio script: fio --rw=randread --size=128G --direct=1 --ioengine=libaio --iodepth=2048 \ --numjobs=32 --bs=4k --group_reporting=1 --group_reporting=1 --runtime=60 \ --loops=10000 --name=job1 --filename=/dev/sdN [1] https://lore.kernel.org/linux-block/20200119071432.18558-6-ming.lei@xxxxxxxxxx/ Cc: Omar Sandoval <osandov@xxxxxx> Cc: Kashyap Desai <kashyap.desai@xxxxxxxxxxxx> Cc: Sumanesh Samanta <sumanesh.samanta@xxxxxxxxxxxx> Cc: Ewan D. Milne <emilne@xxxxxxxxxx> Cc: Hannes Reinecke <hare@xxxxxxx> Signed-off-by: Ming Lei <ming.lei@xxxxxxxxxx> --- drivers/scsi/scsi.c | 2 ++ drivers/scsi/scsi_lib.c | 33 +++++++++++++++------------------ drivers/scsi/scsi_priv.h | 1 + drivers/scsi/scsi_scan.c | 22 ++++++++++++++++++++-- drivers/scsi/scsi_sysfs.c | 2 ++ include/scsi/scsi_device.h | 5 +++-- 6 files changed, 43 insertions(+), 22 deletions(-)
Ah. Now it makes sense why 'no budget' has been set to -1. Ok. Reviewed-by: Hannes Reinecke <hare@xxxxxxx> Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@xxxxxxx +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
