Re: Slow initial resync in RAID6 with 36 SAS drives

[Song Liu <song@xxxxxxxxxx>]

On Tue, Aug 31, 2021 at 10:19 PM Song Liu <song@xxxxxxxxxx> wrote:
>
> On Wed, Aug 25, 2021 at 3:06 AM Marcin Wanat <marcin.wanat@xxxxxxxxx> wrote:
> >
> > On Thu, Aug 19, 2021 at 11:28 AM Marcin Wanat <marcin.wanat@xxxxxxxxx> wrote:
> > >
> > > Sorry, this will be a long email with everything I find to be relevant.
> > > I have a mdraid6 array with 36 hdd SAS drives each able to do
> > > >200MB/s, but I am unable to get more than 38MB/s resync speed on a
> > > fast system (48cores/96GB ram) with no other load.
> >
> > I have done a bit more research on 24 NVMe drives server and found
> > that resync speed bottleneck affect RAID6 with >16 drives:
>
> Sorry for the late response.
>
> This is interesting behavior. I don't really know why this is the case at the
> moment. Let me try to reproduce this first.
>
> Thanks,
> Song

The issue is caused by blk_plug logic. Something like the following should
fix it.

Marcin, could you please give it a try?

Thanks,
Song

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 2c4ac51e54eba..fdb945be85753 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2251,7 +2251,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio)
                else
                        last = list_entry_rq(plug->mq_list.prev);

-               if (request_count >= BLK_MAX_REQUEST_COUNT || (last &&
+               if (request_count >= blk_plug_max_rq_count(plug) || (last &&
                    blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE)) {
                        blk_flush_plug_list(plug, false);
                        trace_block_plug(q);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index b5c033cf5f26f..2e3c07e959c14 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1239,6 +1239,13 @@ extern void blk_start_plug(struct blk_plug *);
 extern void blk_finish_plug(struct blk_plug *);
 extern void blk_flush_plug_list(struct blk_plug *, bool);

+static inline unsigned short blk_plug_max_rq_count(struct blk_plug *plug)
+{
+       if (plug->multiple_queues)
+               return BLK_MAX_REQUEST_COUNT * 4;
+       return BLK_MAX_REQUEST_COUNT;
+}
+
 static inline void blk_flush_plug(struct task_struct *tsk)
 {
        struct blk_plug *plug = tsk->plug;