Hi,
在 2023/06/16 15:31, Ali Gholami Rudi 写道:
Ali Gholami Rudi <aligrudi@xxxxxxxxx> wrote:
Xiao Ni <xni@xxxxxxxxxx> wrote:
On Thu, Jun 15, 2023 at 10:06 PM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
This looks familiar... Perhaps can you try to test with raid10 with
latest mainline kernel? I used to optimize spin_lock for raid10, and I
don't do this for raid1 yet... I can try to do the same thing for raid1
if it's valuable.
I do get improvements with raid10:
Without RAID (writing to /dev/ram0)
READ: IOPS=15.8M BW=60.3GiB/s
WRITE: IOPS= 6.8M BW=27.7GiB/s
RAID1 (writing to /dev/md/test)
READ: IOPS=518K BW=2028MiB/s
WRITE: IOPS=222K BW= 912MiB/s
RAID10 (writing to /dev/md/test)
READ: IOPS=2033k BW=8329MB/s
WRITE: IOPS= 871k BW=3569MB/s
raid10 is about four times faster than raid1.
And this is perf's output for raid10:
+ 97.33% 0.04% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 96.96% 0.02% fio [kernel.kallsyms] [k] do_syscall_64
+ 94.43% 0.03% fio [kernel.kallsyms] [k] __x64_sys_io_submit
- 93.71% 0.04% fio [kernel.kallsyms] [k] io_submit_one
- 93.67% io_submit_one
- 76.03% aio_write
- 75.53% blkdev_write_iter
- 68.95% blk_finish_plug
- flush_plug_callbacks
- 68.93% raid10_unplug
- 64.31% __wake_up_common_lock
- 64.17% _raw_spin_lock_irqsave
native_queued_spin_lock_slowpath
This is unexpected, can you check if your kernel contain following
patch?
commit 460af1f9d9e62acce4a21f9bd00b5bcd5963bcd4
Author: Yu Kuai <yukuai3@xxxxxxxxxx>
Date: Mon May 29 21:11:06 2023 +0800
md/raid1-10: limit the number of plugged bio
If so, can you retest with following patch?
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index d0de8c9fb3cf..6fdd99c3e59a 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -911,7 +911,7 @@ static void flush_pending_writes(struct r10conf *conf)
blk_start_plug(&plug);
raid1_prepare_flush_writes(conf->mddev->bitmap);
- wake_up(&conf->wait_barrier);
+ wake_up_barrier(&conf->wait_barrier);
while (bio) { /* submit pending writes */
struct bio *next = bio->bi_next;
Thanks,
Kuai
- 4.43% submit_bio_noacct_nocheck
- 4.42% __submit_bio
- 2.28% raid10_end_write_request
- 0.82% raid_end_bio_io
0.82% allow_barrier
2.09% brd_submit_bio
- 6.41% __generic_file_write_iter
- 6.08% generic_file_direct_write
- 5.64% __blkdev_direct_IO_async
- 4.72% submit_bio_noacct_nocheck
- 4.69% __submit_bio
- 4.67% md_handle_request
- 4.66% raid10_make_request
2.59% raid10_write_one_disk
- 16.14% aio_read
- 15.07% blkdev_read_iter
- 14.16% __blkdev_direct_IO_async
- 11.36% submit_bio_noacct_nocheck
- 11.17% __submit_bio
- 5.89% md_handle_request
- 5.84% raid10_make_request
+ 4.18% raid10_read_request
- 3.74% raid10_end_read_request
- 2.04% raid_end_bio_io
1.46% allow_barrier
0.55% mempool_free
1.39% brd_submit_bio
- 1.33% bio_iov_iter_get_pages
- 1.00% iov_iter_get_pages
- __iov_iter_get_pages_alloc
- 0.85% get_user_pages_fast
0.75% internal_get_user_pages_fast
0.93% bio_alloc_bioset
0.65% filemap_write_and_wait_range
+ 88.31% 0.86% fio fio [.] thread_main
+ 87.69% 0.00% fio [unknown] [k] 0xffffffffffffffff
+ 87.60% 0.00% fio fio [.] run_threads
+ 87.31% 0.00% fio fio [.] do_io (inlined)
+ 86.60% 0.32% fio libc-2.31.so [.] syscall
+ 85.87% 0.52% fio fio [.] td_io_queue
+ 85.49% 0.18% fio fio [.] fio_libaio_commit
+ 85.45% 0.14% fio fio [.] td_io_commit
+ 85.37% 0.11% fio libaio.so.1.0.1 [.] io_submit
+ 85.35% 0.00% fio fio [.] io_u_submit (inlined)
+ 76.04% 0.01% fio [kernel.kallsyms] [k] aio_write
+ 75.54% 0.01% fio [kernel.kallsyms] [k] blkdev_write_iter
+ 68.96% 0.00% fio [kernel.kallsyms] [k] blk_finish_plug
+ 68.95% 0.00% fio [kernel.kallsyms] [k] flush_plug_callbacks
+ 68.94% 0.13% fio [kernel.kallsyms] [k] raid10_unplug
+ 64.41% 0.03% fio [kernel.kallsyms] [k] _raw_spin_lock_irqsave
+ 64.32% 0.01% fio [kernel.kallsyms] [k] __wake_up_common_lock
+ 64.05% 63.85% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath
+ 21.05% 1.51% fio [kernel.kallsyms] [k] submit_bio_noacct_nocheck
+ 20.97% 1.18% fio [kernel.kallsyms] [k] __blkdev_direct_IO_async
+ 20.29% 0.03% fio [kernel.kallsyms] [k] __submit_bio
+ 16.15% 0.02% fio [kernel.kallsyms] [k] aio_read
..
Thanks,
Ali
.
