Hi,
Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
> > And this is perf's output for raid10:
> >
> > + 97.33% 0.04% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
> > + 96.96% 0.02% fio [kernel.kallsyms] [k] do_syscall_64
> > + 94.43% 0.03% fio [kernel.kallsyms] [k] __x64_sys_io_submit
> > - 93.71% 0.04% fio [kernel.kallsyms] [k] io_submit_one
> > - 93.67% io_submit_one
> > - 76.03% aio_write
> > - 75.53% blkdev_write_iter
> > - 68.95% blk_finish_plug
> > - flush_plug_callbacks
> > - 68.93% raid10_unplug
> > - 64.31% __wake_up_common_lock
> > - 64.17% _raw_spin_lock_irqsave
> > native_queued_spin_lock_slowpath
>
> This is unexpected, can you check if your kernel contain following
> patch?
>
> commit 460af1f9d9e62acce4a21f9bd00b5bcd5963bcd4
> Author: Yu Kuai <yukuai3@xxxxxxxxxx>
> Date: Mon May 29 21:11:06 2023 +0800
>
> md/raid1-10: limit the number of plugged bio
>
> If so, can you retest with following patch?
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index d0de8c9fb3cf..6fdd99c3e59a 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -911,7 +911,7 @@ static void flush_pending_writes(struct r10conf *conf)
>
> blk_start_plug(&plug);
> raid1_prepare_flush_writes(conf->mddev->bitmap);
> - wake_up(&conf->wait_barrier);
> + wake_up_barrier(&conf->wait_barrier);
>
> while (bio) { /* submit pending writes */
> struct bio *next = bio->bi_next;
No, this patch was not present. I applied this one:
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 4fcfcb350d2b..52f0c24128ff 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -905,7 +905,7 @@ static void flush_pending_writes(struct r10conf *conf)
/* flush any pending bitmap writes to disk
* before proceeding w/ I/O */
md_bitmap_unplug(conf->mddev->bitmap);
- wake_up(&conf->wait_barrier);
+ wake_up_barrier(conf);
while (bio) { /* submit pending writes */
struct bio *next = bio->bi_next;
I get almost the same result as before nevertheless:
Without the patch:
READ: IOPS=2033k BW=8329MB/s
WRITE: IOPS= 871k BW=3569MB/s
With the patch:
READ: IOPS=2027K BW=7920MiB/s
WRITE: IOPS= 869K BW=3394MiB/s
Perf:
+ 96.23% 0.04% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 95.86% 0.02% fio [kernel.kallsyms] [k] do_syscall_64
+ 94.30% 0.03% fio [kernel.kallsyms] [k] __x64_sys_io_submit
- 93.63% 0.04% fio [kernel.kallsyms] [k] io_submit_one
- 93.58% io_submit_one
- 76.44% aio_write
- 75.97% blkdev_write_iter
- 70.17% blk_finish_plug
- flush_plug_callbacks
- 70.15% raid10_unplug
- 66.12% __wake_up_common_lock
- 65.97% _raw_spin_lock_irqsave
65.57% native_queued_spin_lock_slowpath
- 3.85% submit_bio_noacct_nocheck
- 3.84% __submit_bio
- 2.09% raid10_end_write_request
- 0.83% raid_end_bio_io
0.82% allow_barrier
1.70% brd_submit_bio
+ 5.59% __generic_file_write_iter
+ 15.71% aio_read
+ 88.38% 0.71% fio fio [.] thread_main
+ 87.89% 0.00% fio [unknown] [k] 0xffffffffffffffff
+ 87.81% 0.00% fio fio [.] run_threads
+ 87.54% 0.00% fio fio [.] do_io (inlined)
+ 86.79% 0.31% fio libc-2.31.so [.] syscall
+ 86.19% 0.54% fio fio [.] td_io_queue
+ 85.79% 0.18% fio fio [.] fio_libaio_commit
+ 85.76% 0.14% fio fio [.] td_io_commit
+ 85.69% 0.14% fio libaio.so.1.0.1 [.] io_submit
+ 85.66% 0.00% fio fio [.] io_u_submit (inlined)
+ 76.45% 0.01% fio [kernel.kallsyms] [k] aio_write
..
