Hi,
在 2023/06/19 13:19, Ali Gholami Rudi 写道:
Hi,
Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
Can you try to test with --bitmap=none, and --assume-clean(or echo
frozen to sync_action), let's see if spin_lock from wait_barrier() and
from md_bitmap_startwrite is bypassed, how performance will be.
I did not notice that it was syncing disks in the background.
It is much better now:
READ: IOPS=1748K BW=6830MiB/s
WRITE: IOPS= 749K BW=2928MiB/s
This looks good, and cost from spin_lock() from raid10_unplug() is about
4%, this can also be avoided, however, performance should not be
improved obviously.
Thanks,
Kuai
Perf's output:
+ 98.04% 0.00% fio [unknown] [k] 0xffffffffffffffff
+ 97.86% 0.00% fio fio [.] 0x000055ed33e52117
+ 94.64% 0.03% fio libc.so.6 [.] syscall
+ 94.54% 0.09% fio [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
+ 94.44% 0.03% fio [kernel.kallsyms] [k] do_syscall_64
+ 94.31% 0.00% fio fio [.] 0x000055ed33e4fceb
+ 94.31% 0.07% fio fio [.] td_io_queue
+ 94.15% 0.03% fio fio [.] td_io_commit
+ 93.77% 0.05% fio libaio.so.1.0.2 [.] io_submit
+ 93.77% 0.00% fio fio [.] 0x000055ed33e74e5e
+ 92.03% 0.04% fio [kernel.kallsyms] [k] __x64_sys_io_submit
- 91.86% 0.05% fio [kernel.kallsyms] [k] io_submit_one
- 91.81% io_submit_one
- 52.43% aio_read
- 51.94% ext4_file_read_iter
- 36.03% iomap_dio_rw
- 36.02% __iomap_dio_rw
- 21.56% iomap_dio_bio_iter
- 17.72% submit_bio_noacct_nocheck
- 17.09% __submit_bio
- 14.81% md_handle_request
- 8.34% raid10_make_request
- 6.96% raid10_read_request
- 2.76% regular_request_wait
2.58% wait_barrier
0.76% read_balance
- 1.07% asm_common_interrupt
- 1.07% common_interrupt
- 1.06% __common_interrupt
+ 1.06% handle_edge_irq
2.22% md_submit_bio
0.51% blk_mq_submit_bio
- 10.68% iomap_iter
- 10.64% ext4_iomap_begin
- 4.07% ext4_map_blocks
- 3.80% ext4_es_lookup_extent
1.77% _raw_read_lock
1.68% _raw_read_unlock
3.51% ext4_set_iomap
+ 0.54% asm_common_interrupt
- 1.21% blk_finish_plug
- 1.20% __blk_flush_plug
- 1.19% blk_mq_flush_plug_list
- 1.15% nvme_queue_rqs
+ 1.01% nvme_prep_rq.part.0
- 6.04% down_read
- 1.73% asm_common_interrupt
- 1.73% common_interrupt
- 1.72% __common_interrupt
- 1.71% handle_edge_irq
- 1.58% handle_irq_event
- 1.57% __handle_irq_event_percpu
- nvme_irq
- 1.46% blk_mq_end_request_batch
0.80% raid10_end_write_request
0.55% raid10_end_read_request
- 5.43% up_read
+ 0.99% asm_common_interrupt
- 3.99% touch_atime
- 3.87% atime_needs_update
+ 0.68% asm_common_interrupt
- 39.00% aio_write
- 38.78% ext4_file_write_iter
- 28.16% iomap_dio_rw
- 28.15% __iomap_dio_rw
- 14.49% iomap_dio_bio_iter
- 12.65% submit_bio_noacct_nocheck
- 12.61% __submit_bio
- 11.54% md_handle_request
- 7.91% raid10_make_request
3.24% raid10_write_one_disk
- 1.21% regular_request_wait
1.13% wait_barrier
1.14% wait_blocked_dev
+ 0.57% asm_common_interrupt
1.04% md_submit_bio
- 8.05% blk_finish_plug
- 8.04% __blk_flush_plug
- 6.42% raid10_unplug
- 4.19% __wake_up_common_lock
- 3.60% _raw_spin_lock_irqsave
3.40% native_queued_spin_lock_slowpath
0.59% _raw_spin_unlock_irqrestore
- 1.00% submit_bio_noacct_nocheck
+ 0.93% blk_mq_submit_bio
- 1.59% blk_mq_flush_plug_list
- 1.02% blk_mq_sched_insert_requests
+ 0.98% blk_mq_try_issue_list_directly
- 4.53% iomap_iter
- 4.51% ext4_iomap_overwrite_begin
- 4.51% ext4_iomap_begin
- 1.69% ext4_map_blocks
- 1.58% ext4_es_lookup_extent
0.72% _raw_read_lock
0.71% _raw_read_unlock
1.54% ext4_set_iomap
- 2.83% up_read
+ 0.92% asm_common_interrupt
- 2.26% down_read
+ 0.57% asm_common_interrupt
- 1.81% ext4_map_blocks
- 1.66% ext4_es_lookup_extent
0.81% _raw_read_lock
0.70% _raw_read_unlock
- 1.67% file_modified_flags
1.43% inode_needs_update_time.part.0
+ 64.19% 0.02% fio [kernel.kallsyms] [k] iomap_dio_rw
+ 64.17% 2.41% fio [kernel.kallsyms] [k] __iomap_dio_rw
+ 52.43% 0.03% fio [kernel.kallsyms] [k] aio_read
+ 51.94% 0.04% fio [kernel.kallsyms] [k] ext4_file_read_iter
Thanks,
Ali
.
