On 9/2/21 6:58 PM, Song Liu wrote:
> On Tue, Aug 31, 2021 at 10:19 PM Song Liu <song@xxxxxxxxxx> wrote:
>>
>> On Wed, Aug 25, 2021 at 3:06 AM Marcin Wanat <marcin.wanat@xxxxxxxxx> wrote:
>>>
>>> On Thu, Aug 19, 2021 at 11:28 AM Marcin Wanat <marcin.wanat@xxxxxxxxx> wrote:
>>>>
>>>> Sorry, this will be a long email with everything I find to be relevant.
>>>> I have a mdraid6 array with 36 hdd SAS drives each able to do
>>>>> 200MB/s, but I am unable to get more than 38MB/s resync speed on a
>>>> fast system (48cores/96GB ram) with no other load.
>>>
>>> I have done a bit more research on 24 NVMe drives server and found
>>> that resync speed bottleneck affect RAID6 with >16 drives:
>>
>> Sorry for the late response.
>>
>> This is interesting behavior. I don't really know why this is the case at the
>> moment. Let me try to reproduce this first.
>>
>> Thanks,
>> Song
>
> The issue is caused by blk_plug logic. Something like the following should
> fix it.
>
> Marcin, could you please give it a try?
>
> Thanks,
> Song
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 2c4ac51e54eba..fdb945be85753 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -2251,7 +2251,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
> else
> last = list_entry_rq(plug->mq_list.prev);
>
> - if (request_count >= BLK_MAX_REQUEST_COUNT || (last &&
> + if (request_count >= blk_plug_max_rq_count(plug) || (last &&
> blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE)) {
> blk_flush_plug_list(plug, false);
> trace_block_plug(q);
> diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
> index b5c033cf5f26f..2e3c07e959c14 100644
> --- a/include/linux/blkdev.h
> +++ b/include/linux/blkdev.h
> @@ -1239,6 +1239,13 @@ extern void blk_start_plug(struct blk_plug *);
> extern void blk_finish_plug(struct blk_plug *);
> extern void blk_flush_plug_list(struct blk_plug *, bool);
>
> +static inline unsigned short blk_plug_max_rq_count(struct blk_plug *plug)
> +{
> + if (plug->multiple_queues)
> + return BLK_MAX_REQUEST_COUNT * 4;
> + return BLK_MAX_REQUEST_COUNT;
> +}
> +
> static inline void blk_flush_plug(struct task_struct *tsk)
> {
> struct blk_plug *plug = tsk->plug;
Just put this in blk-mq.c, there's no reason to put this in blkdev.h. It
could go in block/blk.h, but let's just put it where it's used for now.
Should also have a comment on why we allow more for multiple_queues.
That said, the principle is sound imho.
--
Jens Axboe
