On Thu, Apr 11, 2019 at 5:46 AM Dongsheng Yang
<dongsheng.yang@xxxxxxxxxxxx> wrote:
>
> To improve sequential IO of rbd device, we wish there are
> as more chance to merge IO as possible.
>
> But when the blck_mq_make_request() try to merge with IOs in Software
> queue by blk_mq_sched_bio_merge(), it would find software queue
> is empty at most cases.
>
> The reason is when blk_mq call blk_mq_run_hw_queue() to dispatch requests
> from SQ to driver, it will call .queue_rq() implemented in rbd driver.
> And we only queue a work in rbd_mq_ops->queue_rq() without possible to
> return BLK_STS_DEV_RESOURCE or BLK_STS_RESOURCE. Then the requests in
> SQ will be dequeued in anyway. That means in most case, Software is
> empty when the next request want to do a IO merge.
>
> To improve it, blk_mq provides .get_budget and .put_budget in blk_mq_ops.
> Then request need to get budget at first when dequeue from software queue,
> if it return false, request will be remained in software queue and wait
> for next dispatch.
>
> So this commit introduce an option of busy_threshold in rbd driver, and
> implement .get_budget and .put_budget in rbd_mq_ops. We will increase
> budget_reserved in .get_budget and decrese budget_reserved in .put_budget.
> When the budget_reserved is larger than busy_threshold, return false in
> .get_budget to indecate rbd device is busy now.
>
> Signed-off-by: Dongsheng Yang <dongsheng.yang@xxxxxxxxxxxx>
> ---
> In my testing, we can get about 60% improvement in seq reading and
> similar level in rand reading, with -o busy_threshold=64. (bs=4K iodepth=128 numjobs=1)
Hi Dongsheng,
Have you tried running the same test with -o queue_depth=64? I thought
get_budget was supposed to be a SCSI-specific workaround for a very big
tagset shared between many individual devices capable of handling only
a couple of requests at the same time. For rbd, -o queue_depth controls
the size of the tagset for each individual device, so an extra layer of
budgeting shouldn't be needed?
Also, small block sequential I/O like that is a definite use case for
fancy striping. Have you tried experimenting with different striping
patterns here?
Thanks,
Ilya
