Hi,
在 2023/06/16 10:14, Xiao Ni 写道:
On Thu, Jun 15, 2023 at 10:06 PM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
Hi,
在 2023/06/15 15:54, Ali Gholami Rudi 写道:
Perf output:
Samples: 1M of event 'cycles', Event count (approx.): 1158425235997
Children Self Command Shared Object Symbol
+ 97.98% 0.01% fio fio [.] fio_libaio_commit
+ 97.95% 0.01% fio libaio.so.1.0.1 [.] io_submit
+ 97.85% 0.01% fio [kernel.kallsyms] [k] __x64_sys_io_submit
- 97.82% 0.01% fio [kernel.kallsyms] [k] io_submit_one
- 97.81% io_submit_one
- 54.62% aio_write
- 54.60% blkdev_write_iter
- 36.30% blk_finish_plug
- flush_plug_callbacks
- 36.29% raid1_unplug
- flush_bio_list
- 18.44% submit_bio_noacct
- 18.40% brd_submit_bio
- 18.13% raid1_end_write_request
- 17.94% raid_end_bio_io
- 17.82% __wake_up_common_lock
+ 17.79% _raw_spin_lock_irqsave
- 17.79% __wake_up_common_lock
+ 17.76% _raw_spin_lock_irqsave
+ 18.29% __generic_file_write_iter
- 43.12% aio_read
- 43.07% blkdev_read_iter
- generic_file_read_iter
- 43.04% blkdev_direct_IO
- 42.95% submit_bio_noacct
- 42.23% brd_submit_bio
- 41.91% raid1_end_read_request
- 41.70% raid_end_bio_io
- 41.43% __wake_up_common_lock
+ 41.36% _raw_spin_lock_irqsave
- 0.68% md_submit_bio
0.61% md_handle_request
+ 94.90% 0.00% fio [kernel.kallsyms] [k] __wake_up_common_lock
+ 94.86% 0.22% fio [kernel.kallsyms] [k] _raw_spin_lock_irqsave
+ 94.64% 94.64% fio [kernel.kallsyms] [k] native_queued_spin_lock_slowpath
+ 79.63% 0.02% fio [kernel.kallsyms] [k] submit_bio_noacct
This looks familiar... Perhaps can you try to test with raid10 with
latest mainline kernel? I used to optimize spin_lock for raid10, and I
don't do this for raid1 yet... I can try to do the same thing for raid1
if it's valuable.
Hi Kuai
Which patch?
https://lore.kernel.org/lkml/fbf58ad3-2eff-06df-6426-3b4629408e94@xxxxxxxxx/T/
I have a try on raid10. The results are:
Do your create the arry with 4 ramdisk?
raid10
READ: bw=3711MiB/s (3892MB/s)
WRITE: bw=1590MiB/s (1667MB/s)
raid0
READ: bw=5610MiB/s (5882MB/s)
WRITE: bw=2405MiB/s (2521MB/s)
ram0
READ: bw=5468MiB/s (5734MB/s)
WRITE: bw=2343MiB/s (2457MB/s)
I think spin_lock from fastpath in your machine probably doesn't matter
much. I'll be helpful if Ali can try to test raid10...
Thanks,
Kuai
Because raid10 has a function like raid0. So I did a test on raid0
too. There is a performance gap between raid10 and ram disk too. The
strange thing is that raid0 doesn't have a big performance
improvement.
Regards
Xiao
FIO configuration file:
[global]
name=random reads and writes
ioengine=libaio
direct=1
readwrite=randrw
rwmixread=70
iodepth=64
buffered=0
#filename=/dev/ram0
filename=/dev/dm/test
size=1G
runtime=30
time_based
randrepeat=0
norandommap
refill_buffers
ramp_time=10
bs=4k
numjobs=400
400 is too aggressive, I think spin_lock from fast path is probably
causing the problem, same as I met before for raid10...
Thanks,
Kuai
group_reporting=1
[job1]
.
.
