Hello,
kernel test robot noticed a -72.9% regression of fio.write_iops on:
commit: 574e7779cf583171acb5bf6365047bb0941b387c ("block/mq-deadline: use separate insertion lists")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
testcase: fio-basic
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:
runtime: 300s
disk: 1HDD
fs: xfs
nr_task: 100%
test_size: 128G
rw: write
bs: 4k
ioengine: io_uring
direct: direct
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
| Closes: https://lore.kernel.org/oe-lkp/202401312320.a335db14-oliver.sang@xxxxxxxxx
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240131/202401312320.a335db14-oliver.sang@xxxxxxxxx
=========================================================================================
bs/compiler/cpufreq_governor/direct/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase:
4k/gcc-12/performance/direct/1HDD/xfs/io_uring/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/300s/write/lkp-icl-2sp9/128G/fio-basic
commit:
8f764b91fd ("block/mq-deadline: skip expensive merge lookups if contended")
574e7779cf ("block/mq-deadline: use separate insertion lists")
8f764b91fdf29659 574e7779cf583171acb5bf63650
---------------- ---------------------------
%stddev %change %stddev
\ | \
10031 -68.5% 3157 ± 7% uptime.idle
38.88 -93.1% 2.67 ± 4% iostat.cpu.idle
60.16 +60.2% 96.39 iostat.cpu.iowait
2128969 ± 58% -86.2% 293345 ±114% numa-vmstat.node0.nr_foll_pin_acquired
2127446 ± 58% -86.2% 293180 ±114% numa-vmstat.node0.nr_foll_pin_released
169.33 ± 11% +41.4% 239.50 ± 9% perf-c2c.DRAM.remote
98.67 ± 14% +45.1% 143.17 ± 12% perf-c2c.HITM.remote
96520 ± 3% +50.1% 144877 ± 2% meminfo.Active
96358 ± 3% +50.3% 144805 ± 2% meminfo.Active(anon)
116390 ± 3% +42.2% 165520 ± 2% meminfo.Shmem
38.49 -36.5 2.03 ± 6% mpstat.cpu.all.idle%
60.56 +36.5 97.03 mpstat.cpu.all.iowait%
0.19 -0.0 0.16 mpstat.cpu.all.sys%
0.16 ± 3% -8.5% 0.14 ± 3% turbostat.IPC
75414 ± 5% +278.1% 285115 turbostat.POLL
0.01 +0.1 0.07 turbostat.POLL%
38.89 -93.1% 2.70 ± 4% vmstat.cpu.id
60.14 +60.2% 96.37 vmstat.cpu.wa
92653 -72.9% 25112 vmstat.io.bo
63.11 +340.1% 277.79 vmstat.procs.b
18314 +54.0% 28195 vmstat.system.cs
72218 +5.8% 76405 vmstat.system.in
584892 ± 15% +885.8% 5766120 ±123% sched_debug.cfs_rq:/.load.max
950.75 ± 10% +47.6% 1403 ± 20% sched_debug.cfs_rq:/.load_avg.max
203.10 ± 15% +32.3% 268.69 ± 14% sched_debug.cfs_rq:/.load_avg.stddev
46333 +51.0% 69954 sched_debug.cpu.nr_switches.avg
29986 ± 9% +84.3% 55251 sched_debug.cpu.nr_switches.min
0.00 ± 35% +1e+05% 3.64 ± 9% sched_debug.cpu.nr_uninterruptible.avg
23.03 ± 27% +187.8% 66.28 ± 8% sched_debug.cpu.nr_uninterruptible.max
-18.67 +207.7% -57.44 sched_debug.cpu.nr_uninterruptible.min
7.23 ± 5% +258.7% 25.92 ± 10% sched_debug.cpu.nr_uninterruptible.stddev
24079 ± 3% +50.3% 36201 ± 2% proc-vmstat.nr_active_anon
754201 +1.6% 766356 proc-vmstat.nr_file_pages
3467173 -72.7% 946413 proc-vmstat.nr_foll_pin_acquired
3464673 -72.7% 945922 proc-vmstat.nr_foll_pin_released
11862 +1.6% 12051 proc-vmstat.nr_mapped
29088 ± 3% +42.2% 41375 ± 2% proc-vmstat.nr_shmem
24079 ± 3% +50.3% 36201 ± 2% proc-vmstat.nr_zone_active_anon
1051599 -4.5% 1004608 proc-vmstat.numa_hit
984923 -4.8% 937714 proc-vmstat.numa_local
71376 ± 10% +27.2% 90810 ± 8% proc-vmstat.numa_pte_updates
38600 ± 2% +37.7% 53153 proc-vmstat.pgactivate
1308586 -6.9% 1218899 proc-vmstat.pgalloc_normal
856034 +2.4% 876179 proc-vmstat.pgfault
1224361 -7.3% 1135057 proc-vmstat.pgfree
28164246 -72.9% 7638316 proc-vmstat.pgpgout
0.12 ± 28% +0.8 0.92 ± 9% fio.latency_1000ms%
17.72 ± 3% -10.3 7.39 ± 4% fio.latency_100ms%
0.05 ± 16% +0.2 0.23 ± 5% fio.latency_10ms%
19.15 ± 6% -19.0 0.19 ± 6% fio.latency_20ms%
23.99 ± 3% +7.5 31.50 fio.latency_250ms%
0.01 +0.0 0.03 ± 7% fio.latency_4ms%
5.84 ± 3% +36.8 42.63 fio.latency_500ms%
32.66 ± 6% -29.9 2.73 ± 4% fio.latency_50ms%
0.42 ± 16% +13.9 14.27 fio.latency_750ms%
56330658 -72.9% 15275141 fio.time.file_system_outputs
3139 ± 3% -62.4% 1182 ± 2% fio.time.involuntary_context_switches
8.17 ± 4% -14.3% 7.00 fio.time.percent_of_cpu_this_job_got
1902903 +24.6% 2371272 fio.time.voluntary_context_switches
7041332 -72.9% 1909392 fio.workload
91.62 -72.9% 24.85 fio.write_bw_MBps
2.045e+08 +166.0% 5.439e+08 fio.write_clat_90%_us
3.055e+08 ± 2% +99.1% 6.082e+08 fio.write_clat_95%_us
4.264e+08 ± 3% +76.4% 7.522e+08 fio.write_clat_99%_us
87286745 +257.4% 3.12e+08 fio.write_clat_mean_us
99184810 ± 3% +75.3% 1.739e+08 fio.write_clat_stddev
23454 -72.9% 6361 fio.write_iops
1727 +5.7e+05% 9845156 fio.write_slat_mean_us
7460 ± 39% +4.9e+05% 36525171 fio.write_slat_stddev
1.66 +22.5% 2.04 perf-stat.i.MPKI
0.97 +0.1 1.09 perf-stat.i.branch-miss-rate%
2609625 +7.1% 2794459 ± 4% perf-stat.i.branch-misses
16.66 -0.6 16.05 perf-stat.i.cache-miss-rate%
1901031 +11.6% 2120745 perf-stat.i.cache-misses
11266152 +16.9% 13166895 perf-stat.i.cache-references
18410 +54.3% 28416 perf-stat.i.context-switches
2.10 ± 2% +10.4% 2.32 perf-stat.i.cpi
2.299e+09 ± 2% +5.1% 2.416e+09 perf-stat.i.cpu-cycles
901.94 ± 2% -34.2% 593.78 perf-stat.i.cpu-migrations
1315 ± 2% -12.7% 1148 perf-stat.i.cycles-between-cache-misses
0.04 ± 5% +0.0 0.04 ± 4% perf-stat.i.dTLB-load-miss-rate%
111249 ± 5% +18.1% 131362 ± 4% perf-stat.i.dTLB-load-misses
3.359e+08 -6.7% 3.134e+08 perf-stat.i.dTLB-loads
48568 -14.5% 41547 ± 2% perf-stat.i.dTLB-store-misses
1.737e+08 -11.7% 1.534e+08 perf-stat.i.dTLB-stores
1.215e+09 -5.6% 1.147e+09 perf-stat.i.instructions
0.49 ± 2% -10.2% 0.44 perf-stat.i.ipc
0.04 ± 2% +5.1% 0.04 perf-stat.i.metric.GHz
191.35 +17.5% 224.82 perf-stat.i.metric.K/sec
11.77 -7.1% 10.94 perf-stat.i.metric.M/sec
2459 +2.4% 2518 perf-stat.i.minor-faults
90.60 +3.0 93.58 perf-stat.i.node-load-miss-rate%
452504 +31.3% 594058 perf-stat.i.node-load-misses
51347 ± 3% -9.0% 46745 perf-stat.i.node-loads
209653 ± 3% +17.1% 245593 ± 3% perf-stat.i.node-store-misses
293009 ± 2% +23.4% 361448 perf-stat.i.node-stores
2459 +2.4% 2519 perf-stat.i.page-faults
1.56 +18.2% 1.85 perf-stat.overall.MPKI
1.07 +0.1 1.20 ± 2% perf-stat.overall.branch-miss-rate%
16.87 -0.8 16.11 perf-stat.overall.cache-miss-rate%
1.89 ± 2% +11.3% 2.11 perf-stat.overall.cpi
1209 -5.8% 1139 perf-stat.overall.cycles-between-cache-misses
0.03 ± 5% +0.0 0.04 ± 3% perf-stat.overall.dTLB-load-miss-rate%
0.53 ± 2% -10.2% 0.47 perf-stat.overall.ipc
89.81 +2.9 92.70 perf-stat.overall.node-load-miss-rate%
51848 +248.4% 180654 perf-stat.overall.path-length
2602422 +7.1% 2786733 ± 4% perf-stat.ps.branch-misses
1894822 +11.6% 2113909 perf-stat.ps.cache-misses
11229305 +16.9% 13124260 perf-stat.ps.cache-references
18349 +54.4% 28323 perf-stat.ps.context-switches
2.291e+09 ± 2% +5.1% 2.408e+09 perf-stat.ps.cpu-cycles
898.94 ± 2% -34.2% 591.81 perf-stat.ps.cpu-migrations
110886 ± 5% +18.1% 130940 ± 4% perf-stat.ps.dTLB-load-misses
3.349e+08 -6.7% 3.124e+08 perf-stat.ps.dTLB-loads
48409 -14.4% 41414 ± 2% perf-stat.ps.dTLB-store-misses
1.732e+08 -11.7% 1.529e+08 perf-stat.ps.dTLB-stores
1.211e+09 -5.6% 1.144e+09 perf-stat.ps.instructions
2451 +2.4% 2511 perf-stat.ps.minor-faults
451006 +31.3% 592114 perf-stat.ps.node-load-misses
51197 ± 3% -9.0% 46611 perf-stat.ps.node-loads
208958 ± 3% +17.1% 244787 ± 3% perf-stat.ps.node-store-misses
292047 ± 2% +23.4% 360278 perf-stat.ps.node-stores
2451 +2.4% 2511 perf-stat.ps.page-faults
3.651e+11 -5.5% 3.45e+11 ± 2% perf-stat.total.instructions
0.00 ± 43% +368.2% 0.02 ± 75% perf-sched.sch_delay.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
0.01 -14.3% 0.01 perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xfs_ilock_for_iomap
0.01 ± 42% -35.1% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.io_wq_worker.ret_from_fork.ret_from_fork_asm
0.01 -33.3% 0.01 perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
0.01 ± 19% -27.8% 0.00 ± 10% perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
0.00 ±223% +1912.5% 0.03 ± 94% perf-sched.sch_delay.max.ms.__cond_resched.io_assign_current_work.io_worker_handle_work.io_wq_worker.ret_from_fork
0.00 ± 9% +126.1% 0.01 ± 38% perf-sched.sch_delay.max.ms.__cond_resched.task_work_run.io_run_task_work.io_cqring_wait.__do_sys_io_uring_enter
56.45 ± 3% -32.6% 38.05 perf-sched.total_wait_and_delay.average.ms
41251 ± 4% +99.0% 82087 perf-sched.total_wait_and_delay.count.ms
56.44 ± 3% -32.6% 38.04 perf-sched.total_wait_time.average.ms
10.80 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
376.03 ± 48% -94.4% 21.05 ±223% perf-sched.wait_and_delay.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_iomap_write_unwritten
433.85 ± 59% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
815.52 ± 92% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
486.10 ± 48% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve
240.19 ± 23% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
2.32 ± 10% +22.5% 2.84 ± 5% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.io_wq_worker.ret_from_fork.ret_from_fork_asm
58.50 ± 6% -80.3% 11.54 ± 6% perf-sched.wait_and_delay.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
434.72 ± 12% -49.8% 218.09 ± 4% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
48.19 ± 5% -71.8% 13.60 ±141% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xfs_ilock_for_iomap
37.76 ± 8% -83.0% 6.43 ±223% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.xfs_ilock
36.84 ± 5% +27.8% 47.07 ± 4% perf-sched.wait_and_delay.avg.ms.schedule_timeout.io_wq_worker.ret_from_fork.ret_from_fork_asm
620.47 ± 3% -10.9% 552.59 ± 3% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
64.08 ± 4% -72.5% 17.65 ± 2% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
384.00 -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
18.00 ± 29% -99.1% 0.17 ±223% perf-sched.wait_and_delay.count.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_iomap_write_unwritten
2.50 ± 38% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.down_write.xfs_ilock.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
5.00 ± 65% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
4.00 ± 38% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve
45.17 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
5281 ± 5% -62.8% 1967 ± 12% perf-sched.wait_and_delay.count.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
33.33 ± 13% +110.5% 70.17 ± 3% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
4684 ± 6% -98.2% 82.17 ±141% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xfs_ilock_for_iomap
420.83 ± 8% -99.1% 3.67 ±223% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.xfs_ilock
11733 ± 5% -20.3% 9354 ± 4% perf-sched.wait_and_delay.count.schedule_timeout.io_wq_worker.ret_from_fork.ret_from_fork_asm
622.67 ± 4% -18.7% 506.17 ± 5% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
11804 ± 4% +203.5% 35824 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
1000 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
2061 ± 50% -99.0% 21.05 ±223% perf-sched.wait_and_delay.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_iomap_write_unwritten
782.25 ± 68% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
1601 ± 51% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
1092 ± 35% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve
2843 ± 29% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
511.01 -25.1% 382.56 ± 9% perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.io_wq_worker.ret_from_fork.ret_from_fork_asm
511.46 ± 23% -38.4% 315.16 ± 3% perf-sched.wait_and_delay.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
1060 ± 58% -92.6% 78.42 ±223% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.xfs_ilock
1165 ± 15% +74.8% 2037 ± 17% perf-sched.wait_and_delay.max.ms.schedule_timeout.io_wq_worker.ret_from_fork.ret_from_fork_asm
3997 ± 9% -38.1% 2474 ± 34% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
376.03 ± 48% -92.9% 26.88 ±169% perf-sched.wait_time.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_iomap_write_unwritten
433.84 ± 59% -99.9% 0.43 ±223% perf-sched.wait_time.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
815.52 ± 92% -100.0% 0.06 ±223% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
486.09 ± 48% -100.0% 0.04 ±223% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve
240.19 ± 23% -95.8% 10.16 ± 78% perf-sched.wait_time.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
2.32 ± 10% +22.5% 2.84 ± 5% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.io_wq_worker.ret_from_fork.ret_from_fork_asm
58.50 ± 6% -80.3% 11.53 ± 6% perf-sched.wait_time.avg.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
434.72 ± 12% -49.8% 218.09 ± 4% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
48.18 ± 5% -35.6% 31.03 ± 23% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xfs_ilock_for_iomap
37.74 ± 8% -33.5% 25.11 ± 36% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.xfs_ilock
36.83 ± 5% +27.8% 47.07 ± 4% perf-sched.wait_time.avg.ms.schedule_timeout.io_wq_worker.ret_from_fork.ret_from_fork_asm
620.47 ± 3% -10.9% 552.58 ± 3% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
64.07 ± 4% -72.5% 17.64 ± 2% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
2061 ± 50% -98.7% 26.88 ±169% perf-sched.wait_time.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_iomap_write_unwritten
782.24 ± 68% -99.9% 0.43 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_write.xfs_ilock.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
28.38 ±223% +539.6% 181.50 ± 73% perf-sched.wait_time.max.ms.__cond_resched.io_assign_current_work.io_worker_handle_work.io_wq_worker.ret_from_fork
1601 ± 51% -100.0% 0.06 ±223% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_inode.xfs_iomap_write_unwritten
1092 ± 35% -100.0% 0.04 ±223% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve
2842 ± 29% -99.0% 27.95 ± 81% perf-sched.wait_time.max.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
511.01 -25.1% 382.56 ± 9% perf-sched.wait_time.max.ms.do_task_dead.do_exit.io_wq_worker.ret_from_fork.ret_from_fork_asm
511.46 ± 23% -38.4% 315.16 ± 3% perf-sched.wait_time.max.ms.io_cqring_wait.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
1060 ± 58% -82.5% 185.64 ± 77% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.xfs_ilock
1165 ± 15% +74.8% 2037 ± 17% perf-sched.wait_time.max.ms.schedule_timeout.io_wq_worker.ret_from_fork.ret_from_fork_asm
3997 ± 9% -38.1% 2474 ± 34% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
6.92 ± 10% -3.5 3.42 ± 8% perf-profile.calltrace.cycles-pp.fio_ioring_commit
6.68 ± 11% -3.4 3.31 ± 8% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.fio_ioring_commit
6.41 ± 11% -3.2 3.21 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.fio_ioring_commit
6.34 ± 12% -3.2 3.18 ± 8% perf-profile.calltrace.cycles-pp.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe.fio_ioring_commit
4.54 ± 6% -2.8 1.72 ± 17% perf-profile.calltrace.cycles-pp.iomap_dio_complete_work.process_one_work.worker_thread.kthread.ret_from_fork
4.51 ± 6% -2.8 1.71 ± 17% perf-profile.calltrace.cycles-pp.iomap_dio_complete.iomap_dio_complete_work.process_one_work.worker_thread.kthread
4.73 ± 12% -2.6 2.08 ± 9% perf-profile.calltrace.cycles-pp.io_submit_sqes.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe.fio_ioring_commit
4.20 ± 6% -2.6 1.61 ± 18% perf-profile.calltrace.cycles-pp.xfs_dio_write_end_io.iomap_dio_complete.iomap_dio_complete_work.process_one_work.worker_thread
4.16 ± 6% -2.6 1.57 ± 18% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_dio_write_end_io.iomap_dio_complete.iomap_dio_complete_work.process_one_work
5.32 ± 7% -2.5 2.80 ± 11% perf-profile.calltrace.cycles-pp.do_exit.io_wq_worker.ret_from_fork.ret_from_fork_asm
4.17 ± 15% -2.5 1.72 ± 10% perf-profile.calltrace.cycles-pp.io_issue_sqe.io_submit_sqes.__do_sys_io_uring_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.05 ± 15% -2.4 1.64 ± 11% perf-profile.calltrace.cycles-pp.io_write.io_issue_sqe.io_submit_sqes.__do_sys_io_uring_enter.do_syscall_64
6.65 ± 7% -2.4 4.28 ± 8% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
3.84 ± 16% -2.3 1.56 ± 10% perf-profile.calltrace.cycles-pp.xfs_file_write_iter.io_write.io_issue_sqe.io_submit_sqes.__do_sys_io_uring_enter
4.30 ± 10% -2.3 2.03 ± 18% perf-profile.calltrace.cycles-pp.exit_notify.do_exit.io_wq_worker.ret_from_fork.ret_from_fork_asm
3.74 ± 16% -2.2 1.52 ± 10% perf-profile.calltrace.cycles-pp.xfs_file_dio_write_aligned.xfs_file_write_iter.io_write.io_issue_sqe.io_submit_sqes
2.58 ± 8% -1.2 1.35 ± 12% perf-profile.calltrace.cycles-pp.release_task.exit_notify.do_exit.io_wq_worker.ret_from_fork
1.68 ± 18% -1.2 0.50 ± 81% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.exit_notify.do_exit.io_wq_worker
1.71 ± 18% -1.1 0.60 ± 58% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.exit_notify.do_exit.io_wq_worker.ret_from_fork
1.98 ± 10% -1.0 0.96 ± 20% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.release_task.exit_notify.do_exit.io_wq_worker
1.92 ± 11% -1.0 0.93 ± 20% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.release_task.exit_notify.do_exit
1.36 ± 11% -0.7 0.67 ± 6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_dio_write_end_io.iomap_dio_complete.iomap_dio_complete_work
1.42 ± 5% -0.6 0.82 ± 4% perf-profile.calltrace.cycles-pp.blk_update_request.scsi_end_request.scsi_io_completion.complete_cmd_fusion.megasas_isr_fusion
1.04 ± 11% -0.5 0.56 ± 5% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_dio_write_end_io.iomap_dio_complete
0.96 ± 10% -0.3 0.68 ± 6% perf-profile.calltrace.cycles-pp.iomap_dio_bio_end_io.blk_update_request.scsi_end_request.scsi_io_completion.complete_cmd_fusion
0.74 ± 7% +0.2 0.93 ± 10% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.76 ± 7% +0.2 0.97 ± 10% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.82 ± 7% +0.3 1.14 ± 11% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
0.87 ± 6% +0.3 1.19 ± 11% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.io_wq_worker
1.76 ± 3% +0.6 2.32 ± 3% perf-profile.calltrace.cycles-pp.scsi_end_request.scsi_io_completion.complete_cmd_fusion.megasas_isr_fusion.__handle_irq_event_percpu
1.76 ± 3% +0.6 2.32 ± 3% perf-profile.calltrace.cycles-pp.scsi_io_completion.complete_cmd_fusion.megasas_isr_fusion.__handle_irq_event_percpu.handle_irq_event
0.00 +0.6 0.57 ± 7% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
0.00 +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single
0.00 +0.6 0.63 ± 10% perf-profile.calltrace.cycles-pp.__blk_mq_free_request.scsi_end_request.scsi_io_completion.complete_cmd_fusion.megasas_isr_fusion
0.00 +0.7 0.69 ± 9% perf-profile.calltrace.cycles-pp.blk_mq_run_hw_queues.scsi_end_request.scsi_io_completion.complete_cmd_fusion.megasas_isr_fusion
0.09 ±223% +0.7 0.82 ± 8% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single
0.39 ± 71% +0.7 1.13 ± 7% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
0.00 +0.8 0.77 ± 12% perf-profile.calltrace.cycles-pp.xfs_vn_update_time.kiocb_modified.xfs_file_write_checks.xfs_file_dio_write_aligned.xfs_file_write_iter
0.00 +0.8 0.80 ± 13% perf-profile.calltrace.cycles-pp.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn
0.00 +0.8 0.81 ± 11% perf-profile.calltrace.cycles-pp.kiocb_modified.xfs_file_write_checks.xfs_file_dio_write_aligned.xfs_file_write_iter.io_write
0.58 ± 46% +0.8 1.39 ± 9% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt
0.00 +0.8 0.82 ± 20% perf-profile.calltrace.cycles-pp.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue.blk_mq_get_tag
0.00 +0.8 0.83 ± 20% perf-profile.calltrace.cycles-pp.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue.blk_mq_get_tag.__blk_mq_alloc_requests
0.00 +0.8 0.83 ± 20% perf-profile.calltrace.cycles-pp.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio
0.60 ± 46% +0.8 1.44 ± 9% perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter
0.00 +0.8 0.84 ± 10% perf-profile.calltrace.cycles-pp.xfs_file_write_checks.xfs_file_dio_write_aligned.xfs_file_write_iter.io_write.io_issue_sqe
0.73 ± 6% +0.9 1.60 ± 9% perf-profile.calltrace.cycles-pp.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn.process_one_work.worker_thread
0.73 ± 6% +0.9 1.60 ± 9% perf-profile.calltrace.cycles-pp.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn.process_one_work.worker_thread.kthread
0.73 ± 6% +0.9 1.60 ± 9% perf-profile.calltrace.cycles-pp.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_work_fn.process_one_work
0.73 ± 6% +0.9 1.60 ± 9% perf-profile.calltrace.cycles-pp.blk_mq_run_work_fn.process_one_work.worker_thread.kthread.ret_from_fork
1.99 ± 3% +0.9 2.87 ± 2% perf-profile.calltrace.cycles-pp.complete_cmd_fusion.megasas_isr_fusion.__handle_irq_event_percpu.handle_irq_event.handle_edge_irq
0.79 ± 17% +0.9 1.69 ± 8% perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
0.00 +0.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.blk_mq_run_hw_queue.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.submit_bio_noacct_nocheck
2.01 ± 3% +0.9 2.91 ± 2% perf-profile.calltrace.cycles-pp.__handle_irq_event_percpu.handle_irq_event.handle_edge_irq.__common_interrupt.common_interrupt
2.00 ± 3% +0.9 2.90 ± 2% perf-profile.calltrace.cycles-pp.megasas_isr_fusion.__handle_irq_event_percpu.handle_irq_event.handle_edge_irq.__common_interrupt
2.03 ± 3% +0.9 2.95 ± 2% perf-profile.calltrace.cycles-pp.handle_irq_event.handle_edge_irq.__common_interrupt.common_interrupt.asm_common_interrupt
2.05 ± 3% +0.9 2.98 ± 3% perf-profile.calltrace.cycles-pp.handle_edge_irq.__common_interrupt.common_interrupt.asm_common_interrupt.acpi_safe_halt
2.06 ± 3% +0.9 3.00 ± 3% perf-profile.calltrace.cycles-pp.__common_interrupt.common_interrupt.asm_common_interrupt.acpi_safe_halt.acpi_idle_enter
0.00 +1.0 0.97 ± 6% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.io_schedule
0.00 +1.0 0.98 ± 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.io_schedule.blk_mq_get_tag
2.09 ± 3% +1.0 3.10 ± 2% perf-profile.calltrace.cycles-pp.common_interrupt.asm_common_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
2.10 ± 4% +1.0 3.12 ± 2% perf-profile.calltrace.cycles-pp.asm_common_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
0.00 +1.5 1.52 ± 13% perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.blk_mq_get_tag.__blk_mq_alloc_requests
0.00 +1.6 1.61 ± 12% perf-profile.calltrace.cycles-pp.schedule.io_schedule.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio
0.00 +1.7 1.68 ± 12% perf-profile.calltrace.cycles-pp.io_schedule.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.submit_bio_noacct_nocheck
0.50 ± 47% +1.7 2.23 ± 8% perf-profile.calltrace.cycles-pp.__schedule.schedule.worker_thread.kthread.ret_from_fork
0.50 ± 47% +1.7 2.25 ± 7% perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
0.17 ±141% +1.8 1.94 ± 7% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.worker_thread.kthread
0.08 ±223% +1.8 1.88 ± 7% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.worker_thread
0.00 +2.0 2.00 ± 5% perf-profile.calltrace.cycles-pp.__irqentry_text_start.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
0.61 ± 9% +2.2 2.80 ± 6% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance
0.66 ± 9% +2.4 3.07 ± 6% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
0.67 ± 8% +2.4 3.12 ± 6% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
0.70 ± 7% +2.6 3.30 ± 7% perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
2.47 ± 6% +2.9 5.34 ± 6% perf-profile.calltrace.cycles-pp.iomap_dio_bio_iter.__iomap_dio_rw.iomap_dio_rw.xfs_file_dio_write_aligned.xfs_file_write_iter
3.37 ± 11% +3.3 6.66 ± 8% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
1.27 ± 10% +3.5 4.82 ± 6% perf-profile.calltrace.cycles-pp.submit_bio_noacct_nocheck.iomap_dio_bio_iter.__iomap_dio_rw.iomap_dio_rw.xfs_file_dio_write_aligned
1.02 ± 31% +3.8 4.78 ± 6% perf-profile.calltrace.cycles-pp.blk_mq_submit_bio.submit_bio_noacct_nocheck.iomap_dio_bio_iter.__iomap_dio_rw.iomap_dio_rw
0.00 +4.0 4.04 ± 7% perf-profile.calltrace.cycles-pp.blk_mq_get_tag.__blk_mq_alloc_requests.blk_mq_submit_bio.submit_bio_noacct_nocheck.iomap_dio_bio_iter
0.00 +4.2 4.15 ± 7% perf-profile.calltrace.cycles-pp.__blk_mq_alloc_requests.blk_mq_submit_bio.submit_bio_noacct_nocheck.iomap_dio_bio_iter.__iomap_dio_rw
0.86 ± 10% +5.1 5.97 ± 3% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
63.38 ± 2% +5.7 69.08 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
66.58 ± 2% +5.8 72.38 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
63.90 ± 2% +5.9 69.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
68.18 ± 2% +5.9 74.13 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
68.18 ± 2% +5.9 74.12 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
68.04 ± 2% +6.0 74.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
69.10 ± 2% +6.3 75.35 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
9.01 ± 7% -4.0 5.00 ± 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
8.73 ± 7% -3.8 4.88 ± 5% perf-profile.children.cycles-pp.do_syscall_64
6.94 ± 9% -3.7 3.26 ± 7% perf-profile.children.cycles-pp.__do_sys_io_uring_enter
6.96 ± 10% -3.5 3.44 ± 7% perf-profile.children.cycles-pp.fio_ioring_commit
6.32 ± 28% -3.4 2.90 ± 16% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
4.56 ± 6% -2.8 1.72 ± 17% perf-profile.children.cycles-pp.iomap_dio_complete
4.55 ± 6% -2.8 1.72 ± 17% perf-profile.children.cycles-pp.iomap_dio_complete_work
4.74 ± 12% -2.7 2.09 ± 9% perf-profile.children.cycles-pp.io_submit_sqes
4.17 ± 6% -2.6 1.57 ± 18% perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
4.21 ± 6% -2.6 1.61 ± 18% perf-profile.children.cycles-pp.xfs_dio_write_end_io
5.50 ± 7% -2.5 2.95 ± 10% perf-profile.children.cycles-pp.do_exit
6.66 ± 7% -2.4 4.28 ± 8% perf-profile.children.cycles-pp.process_one_work
4.30 ± 10% -2.3 2.04 ± 18% perf-profile.children.cycles-pp.exit_notify
3.69 ± 13% -2.1 1.64 ± 25% perf-profile.children.cycles-pp.queued_write_lock_slowpath
1.64 ± 19% -1.3 0.32 ± 16% perf-profile.children.cycles-pp.iomap_iter
1.53 ± 21% -1.3 0.27 ± 14% perf-profile.children.cycles-pp.xfs_direct_write_iomap_begin
2.59 ± 8% -1.2 1.35 ± 12% perf-profile.children.cycles-pp.release_task
1.48 ±121% -1.1 0.34 ± 19% perf-profile.children.cycles-pp.dd_insert_requests
1.89 ± 94% -1.1 0.82 ± 9% perf-profile.children.cycles-pp.blk_finish_plug
1.88 ± 94% -1.1 0.82 ± 9% perf-profile.children.cycles-pp.__blk_flush_plug
1.85 ± 96% -1.0 0.81 ± 9% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
1.84 ± 97% -1.0 0.80 ± 9% perf-profile.children.cycles-pp.blk_mq_dispatch_plug_list
1.46 ± 3% -1.0 0.43 ± 39% perf-profile.children.cycles-pp.xfs_trans_alloc_inode
0.95 ± 11% -0.9 0.10 ± 25% perf-profile.children.cycles-pp.xfs_ilock_for_iomap
0.83 ± 9% -0.7 0.14 ± 18% perf-profile.children.cycles-pp.down_read
0.81 ± 5% -0.6 0.20 ±107% perf-profile.children.cycles-pp.xfs_ilock
0.78 ± 5% -0.6 0.18 ±117% perf-profile.children.cycles-pp.down_write
0.69 ± 24% -0.6 0.10 ± 37% perf-profile.children.cycles-pp.fio_ioring_getevents
1.44 ± 6% -0.6 0.86 ± 3% perf-profile.children.cycles-pp.blk_update_request
1.51 ± 13% -0.5 0.97 ± 21% perf-profile.children.cycles-pp.io_cqring_wait
0.67 ± 8% -0.5 0.15 ± 15% perf-profile.children.cycles-pp.xfs_iunlock
0.76 ± 9% -0.5 0.25 ± 22% perf-profile.children.cycles-pp.bio_iov_iter_get_pages
0.75 ± 8% -0.5 0.25 ± 22% perf-profile.children.cycles-pp.__bio_iov_iter_get_pages
0.64 ± 24% -0.5 0.14 ± 14% perf-profile.children.cycles-pp.__io_run_local_work
0.69 ± 8% -0.5 0.23 ± 25% perf-profile.children.cycles-pp.iov_iter_extract_pages
0.67 ± 9% -0.5 0.22 ± 25% perf-profile.children.cycles-pp.pin_user_pages_fast
0.66 ± 9% -0.4 0.22 ± 25% perf-profile.children.cycles-pp.internal_get_user_pages_fast
0.44 ± 16% -0.4 0.04 ± 73% perf-profile.children.cycles-pp.up_write
0.84 ± 13% -0.4 0.45 ± 33% perf-profile.children.cycles-pp.xfs_bmapi_write
1.52 ± 10% -0.4 1.15 ± 6% perf-profile.children.cycles-pp.__xfs_trans_commit
0.55 ± 11% -0.3 0.20 ± 26% perf-profile.children.cycles-pp.lockless_pages_from_mm
0.78 ± 14% -0.3 0.45 ± 9% perf-profile.children.cycles-pp.dd_bio_merge
0.64 ± 19% -0.3 0.34 ± 8% perf-profile.children.cycles-pp.blk_mq_sched_try_merge
0.46 ± 15% -0.3 0.17 ± 23% perf-profile.children.cycles-pp.gup_pgd_range
0.98 ± 10% -0.3 0.72 ± 5% perf-profile.children.cycles-pp.iomap_dio_bio_end_io
0.51 ± 24% -0.3 0.24 ± 14% perf-profile.children.cycles-pp.kmem_cache_free
0.40 ± 16% -0.3 0.14 ± 24% perf-profile.children.cycles-pp.__slab_free
0.38 ± 16% -0.2 0.14 ± 23% perf-profile.children.cycles-pp.gup_pte_range
0.32 ± 75% -0.2 0.10 ± 25% perf-profile.children.cycles-pp.xfs_bmapi_read
0.60 ± 9% -0.2 0.39 ± 21% perf-profile.children.cycles-pp.xfs_trans_alloc
0.38 ± 15% -0.2 0.17 ± 15% perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
0.27 ± 31% -0.2 0.06 ± 52% perf-profile.children.cycles-pp.xfs_ilock_iocb_for_write
0.31 ± 18% -0.2 0.11 ± 27% perf-profile.children.cycles-pp.try_grab_folio
0.24 ± 19% -0.2 0.05 ± 54% perf-profile.children.cycles-pp.fio_gettime
0.34 ± 16% -0.2 0.15 ± 6% perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
0.25 ± 12% -0.2 0.06 ± 59% perf-profile.children.cycles-pp.__io_req_task_work_add
0.27 ± 8% -0.2 0.09 ± 12% perf-profile.children.cycles-pp.__bio_release_pages
0.48 ± 11% -0.2 0.31 ± 20% perf-profile.children.cycles-pp.xfs_trans_reserve
0.20 ± 14% -0.2 0.03 ±100% perf-profile.children.cycles-pp.xfs_ilock_nowait
0.31 ± 15% -0.2 0.16 ± 23% perf-profile.children.cycles-pp.kmem_cache_alloc
0.24 ± 15% -0.2 0.09 ± 24% perf-profile.children.cycles-pp.kfree
0.35 ± 44% -0.2 0.20 ± 20% perf-profile.children.cycles-pp.llist_reverse_order
0.24 ± 11% -0.1 0.09 ± 23% perf-profile.children.cycles-pp.xlog_ticket_alloc
0.17 ± 76% -0.1 0.02 ± 99% perf-profile.children.cycles-pp.io_req_rw_complete
0.42 ± 9% -0.1 0.27 ± 22% perf-profile.children.cycles-pp.xfs_log_reserve
0.23 ± 10% -0.1 0.09 ± 29% perf-profile.children.cycles-pp.bio_alloc_bioset
0.17 ± 23% -0.1 0.03 ±100% perf-profile.children.cycles-pp.down_read_trylock
0.47 ± 19% -0.1 0.33 ± 20% perf-profile.children.cycles-pp.io_queue_iowq
0.16 ± 30% -0.1 0.04 ± 75% perf-profile.children.cycles-pp.get_io_u
0.22 ± 11% -0.1 0.11 ± 8% perf-profile.children.cycles-pp.up_read
0.16 ± 33% -0.1 0.05 ± 76% perf-profile.children.cycles-pp.xfs_trans_run_precommits
0.16 ± 8% -0.1 0.05 ± 71% perf-profile.children.cycles-pp.__cond_resched
0.15 ± 26% -0.1 0.04 ± 72% perf-profile.children.cycles-pp.__io_submit_flush_completions
0.27 ± 18% -0.1 0.16 ± 5% perf-profile.children.cycles-pp.__blk_bios_map_sg
0.12 ± 26% -0.1 0.03 ±100% perf-profile.children.cycles-pp.io_free_batch_list
0.20 ± 22% -0.1 0.10 ± 14% perf-profile.children.cycles-pp.xfs_trans_ijoin
0.12 ± 33% -0.1 0.03 ±100% perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
0.27 ± 17% -0.1 0.18 ± 10% perf-profile.children.cycles-pp.__blk_rq_map_sg
0.18 ± 16% -0.1 0.09 ± 42% perf-profile.children.cycles-pp.memset_orig
0.32 ± 17% -0.1 0.24 ± 15% perf-profile.children.cycles-pp.scsi_alloc_sgtables
0.21 ± 15% -0.1 0.13 ± 17% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
0.10 ± 27% -0.1 0.03 ±102% perf-profile.children.cycles-pp.mempool_alloc
0.33 ± 16% -0.1 0.26 ± 13% perf-profile.children.cycles-pp.sd_setup_read_write_cmnd
0.09 ± 17% -0.1 0.03 ±100% perf-profile.children.cycles-pp.get_random_u32
0.18 ± 18% -0.1 0.12 ± 28% perf-profile.children.cycles-pp.run_timer_softirq
0.12 ± 13% -0.1 0.06 ± 48% perf-profile.children.cycles-pp.xfs_iext_get_extent
0.10 ± 29% -0.1 0.04 ± 73% perf-profile.children.cycles-pp.bio_associate_blkg
0.01 ±223% +0.1 0.06 ± 15% perf-profile.children.cycles-pp.xfs_ag_block_count
0.12 ± 30% +0.1 0.18 ± 13% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.finish_wait
0.03 ±141% +0.1 0.08 ± 38% perf-profile.children.cycles-pp.__enqueue_entity
0.12 ± 13% +0.1 0.19 ± 12% perf-profile.children.cycles-pp.set_next_entity
0.00 +0.1 0.07 ± 20% perf-profile.children.cycles-pp.try_to_grab_pending
0.12 ± 23% +0.1 0.20 ± 30% perf-profile.children.cycles-pp.wakeup_preempt
0.02 ±142% +0.1 0.10 ± 40% perf-profile.children.cycles-pp.megasas_build_ldio_fusion
0.16 ± 30% +0.1 0.25 ± 18% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.12 ± 32% +0.1 0.22 ± 12% perf-profile.children.cycles-pp.llist_add_batch
0.11 ± 9% +0.1 0.22 ± 21% perf-profile.children.cycles-pp.rb_erase
0.13 ± 18% +0.1 0.24 ± 23% perf-profile.children.cycles-pp.elv_attempt_insert_merge
0.17 ± 15% +0.1 0.28 ± 15% perf-profile.children.cycles-pp.scsi_dispatch_cmd
0.08 ± 58% +0.1 0.19 ± 13% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.07 ± 20% +0.1 0.19 ± 19% perf-profile.children.cycles-pp.xas_load
0.00 +0.1 0.12 ± 26% perf-profile.children.cycles-pp.sbitmap_finish_wait
0.00 +0.1 0.13 ± 16% perf-profile.children.cycles-pp.elv_rb_del
0.56 ± 9% +0.1 0.69 ± 7% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.28 ± 13% +0.1 0.41 ± 6% perf-profile.children.cycles-pp.queue_work_on
0.00 +0.1 0.14 ± 28% perf-profile.children.cycles-pp.sbitmap_queue_get_shallow
0.06 ± 50% +0.1 0.20 ± 20% perf-profile.children.cycles-pp.kblockd_mod_delayed_work_on
0.06 ± 50% +0.1 0.20 ± 20% perf-profile.children.cycles-pp.mod_delayed_work_on
0.11 ± 14% +0.2 0.27 ± 11% perf-profile.children.cycles-pp.xas_find
0.02 ±142% +0.2 0.18 ± 21% perf-profile.children.cycles-pp.dd_has_work
0.24 ± 20% +0.2 0.40 ± 15% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.10 ± 24% +0.2 0.27 ± 13% perf-profile.children.cycles-pp._find_next_zero_bit
0.09 ± 25% +0.2 0.26 ± 14% perf-profile.children.cycles-pp.elv_rqhash_find
0.37 ± 14% +0.2 0.54 ± 14% perf-profile.children.cycles-pp.idle_cpu
0.22 ± 11% +0.2 0.40 ± 9% perf-profile.children.cycles-pp._find_next_and_bit
0.55 ± 10% +0.2 0.75 ± 11% perf-profile.children.cycles-pp.update_load_avg
0.00 +0.2 0.21 ± 19% perf-profile.children.cycles-pp.prepare_to_wait_exclusive
0.10 ± 20% +0.2 0.31 ± 15% perf-profile.children.cycles-pp.__dd_dispatch_request
0.66 ± 9% +0.2 0.88 ± 9% perf-profile.children.cycles-pp.update_rq_clock
0.01 ±223% +0.2 0.24 ± 27% perf-profile.children.cycles-pp.__sbitmap_weight
0.01 ±223% +0.2 0.25 ± 25% perf-profile.children.cycles-pp.sbitmap_weight
0.07 ± 28% +0.3 0.32 ± 15% perf-profile.children.cycles-pp.sbitmap_get_shallow
0.47 ± 21% +0.3 0.72 ± 12% perf-profile.children.cycles-pp.enqueue_entity
0.28 ± 10% +0.3 0.54 ± 4% perf-profile.children.cycles-pp.__queue_work
0.77 ± 7% +0.3 1.04 ± 8% perf-profile.children.cycles-pp.schedule_idle
1.04 ± 5% +0.3 1.32 ± 7% perf-profile.children.cycles-pp.try_to_wake_up
0.22 ± 14% +0.3 0.50 ± 6% perf-profile.children.cycles-pp.kick_pool
0.12 ± 27% +0.3 0.40 ± 12% perf-profile.children.cycles-pp.xa_find_after
0.25 ± 10% +0.3 0.53 ± 9% perf-profile.children.cycles-pp.cpu_util
0.55 ± 12% +0.3 0.88 ± 9% perf-profile.children.cycles-pp.xfs_file_write_checks
0.60 ± 17% +0.3 0.94 ± 6% perf-profile.children.cycles-pp.activate_task
0.56 ± 17% +0.4 0.92 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair
0.42 ± 14% +0.4 0.82 ± 10% perf-profile.children.cycles-pp.kiocb_modified
0.00 +0.4 0.41 ± 18% perf-profile.children.cycles-pp.__dd_do_insert
0.31 ± 13% +0.5 0.77 ± 12% perf-profile.children.cycles-pp.xfs_vn_update_time
0.00 +0.5 0.48 ± 7% perf-profile.children.cycles-pp.autoremove_wake_function
0.23 ± 11% +0.5 0.72 ± 8% perf-profile.children.cycles-pp.blk_mq_run_hw_queues
0.65 ± 16% +0.5 1.16 ± 6% perf-profile.children.cycles-pp.ttwu_do_activate
0.65 ± 18% +0.5 1.18 ± 8% perf-profile.children.cycles-pp.sched_ttwu_pending
0.18 ± 10% +0.6 0.74 ± 3% perf-profile.children.cycles-pp.sbitmap_find_bit
0.00 +0.6 0.56 ± 9% perf-profile.children.cycles-pp.__wake_up_common
0.00 +0.6 0.56 ± 9% perf-profile.children.cycles-pp.__wake_up
0.00 +0.6 0.58 ± 12% perf-profile.children.cycles-pp.sbitmap_queue_wake_up
1.80 ± 4% +0.6 2.40 ± 2% perf-profile.children.cycles-pp.scsi_end_request
1.80 ± 4% +0.6 2.41 ± 2% perf-profile.children.cycles-pp.scsi_io_completion
0.41 ± 6% +0.6 1.02 ± 6% perf-profile.children.cycles-pp.__irqentry_text_start
0.02 ±145% +0.6 0.64 ± 11% perf-profile.children.cycles-pp.sbitmap_queue_clear
0.16 ± 22% +0.6 0.80 ± 19% perf-profile.children.cycles-pp.sbitmap_get
0.10 ± 25% +0.6 0.74 ± 11% perf-profile.children.cycles-pp.__blk_mq_free_request
0.82 ± 17% +0.7 1.47 ± 9% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.83 ± 16% +0.7 1.49 ± 8% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.17 ± 36% +0.8 0.94 ± 11% perf-profile.children.cycles-pp.dd_dispatch_request
0.95 ± 15% +0.8 1.73 ± 7% perf-profile.children.cycles-pp.sysvec_call_function_single
0.73 ± 6% +0.9 1.61 ± 9% perf-profile.children.cycles-pp.blk_mq_run_work_fn
2.04 ± 4% +0.9 2.98 perf-profile.children.cycles-pp.complete_cmd_fusion
2.05 ± 3% +1.0 3.01 perf-profile.children.cycles-pp.megasas_isr_fusion
2.06 ± 4% +1.0 3.02 perf-profile.children.cycles-pp.__handle_irq_event_percpu
2.08 ± 3% +1.0 3.05 perf-profile.children.cycles-pp.handle_irq_event
2.10 ± 3% +1.0 3.09 ± 2% perf-profile.children.cycles-pp.handle_edge_irq
2.11 ± 3% +1.0 3.10 ± 2% perf-profile.children.cycles-pp.__common_interrupt
2.14 ± 4% +1.1 3.21 perf-profile.children.cycles-pp.common_interrupt
0.20 ± 28% +1.1 1.28 ± 12% perf-profile.children.cycles-pp.scsi_mq_get_budget
2.14 ± 4% +1.1 3.24 perf-profile.children.cycles-pp.asm_common_interrupt
0.29 ± 32% +1.3 1.58 ± 9% perf-profile.children.cycles-pp.blk_mq_run_hw_queue
1.85 ± 6% +1.8 3.66 ± 5% perf-profile.children.cycles-pp.update_sg_lb_stats
2.12 ± 4% +1.8 3.97 ± 5% perf-profile.children.cycles-pp.update_sd_lb_stats
2.18 ± 4% +1.9 4.03 ± 5% perf-profile.children.cycles-pp.find_busiest_group
2.79 ± 4% +1.9 4.73 ± 6% perf-profile.children.cycles-pp.load_balance
2.33 ± 12% +2.0 4.30 ± 7% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.00 +2.0 2.01 ± 12% perf-profile.children.cycles-pp.io_schedule
0.98 ± 12% +2.0 3.00 ± 7% perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
0.98 ± 13% +2.0 3.01 ± 7% perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
0.98 ± 13% +2.0 3.02 ± 7% perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
2.09 ± 5% +2.3 4.43 ± 6% perf-profile.children.cycles-pp.newidle_balance
2.26 ± 4% +2.5 4.75 ± 5% perf-profile.children.cycles-pp.pick_next_task_fair
2.48 ± 6% +2.9 5.34 ± 6% perf-profile.children.cycles-pp.iomap_dio_bio_iter
2.90 ± 7% +2.9 5.81 ± 5% perf-profile.children.cycles-pp.schedule
3.83 ± 5% +3.1 6.90 ± 4% perf-profile.children.cycles-pp.__schedule
1.27 ± 10% +3.6 4.82 ± 6% perf-profile.children.cycles-pp.submit_bio_noacct_nocheck
1.18 ± 9% +3.6 4.78 ± 6% perf-profile.children.cycles-pp.blk_mq_submit_bio
0.09 ± 17% +3.9 4.04 ± 7% perf-profile.children.cycles-pp.blk_mq_get_tag
0.15 ± 15% +4.0 4.15 ± 7% perf-profile.children.cycles-pp.__blk_mq_alloc_requests
0.88 ± 10% +5.3 6.14 ± 3% perf-profile.children.cycles-pp.poll_idle
64.25 ± 2% +5.9 70.18 perf-profile.children.cycles-pp.cpuidle_enter
68.18 ± 2% +5.9 74.13 perf-profile.children.cycles-pp.start_secondary
63.95 ± 2% +6.0 69.91 perf-profile.children.cycles-pp.cpuidle_enter_state
67.55 ± 2% +6.0 73.59 perf-profile.children.cycles-pp.cpuidle_idle_call
69.08 ± 2% +6.3 75.34 perf-profile.children.cycles-pp.do_idle
69.10 ± 2% +6.3 75.35 perf-profile.children.cycles-pp.cpu_startup_entry
69.10 ± 2% +6.3 75.35 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
6.32 ± 28% -3.4 2.89 ± 16% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.46 ± 16% -0.3 0.12 ± 16% perf-profile.self.cycles-pp.__iomap_dio_rw
0.39 ± 18% -0.3 0.14 ± 25% perf-profile.self.cycles-pp.__slab_free
0.45 ± 20% -0.2 0.23 ± 13% perf-profile.self.cycles-pp.iomap_dio_bio_end_io
0.23 ± 20% -0.2 0.04 ± 77% perf-profile.self.cycles-pp.fio_gettime
0.29 ± 19% -0.2 0.10 ± 27% perf-profile.self.cycles-pp.try_grab_folio
0.29 ± 40% -0.2 0.11 ± 30% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.26 ± 11% -0.2 0.09 ± 51% perf-profile.self.cycles-pp.try_to_wake_up
0.16 ± 24% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.down_read_trylock
0.22 ± 9% -0.1 0.11 ± 10% perf-profile.self.cycles-pp.up_read
0.18 ± 23% -0.1 0.08 ± 19% perf-profile.self.cycles-pp.kmem_cache_free
0.14 ± 30% -0.1 0.04 ± 77% perf-profile.self.cycles-pp.get_io_u
0.26 ± 18% -0.1 0.16 ± 5% perf-profile.self.cycles-pp.__blk_bios_map_sg
0.14 ± 20% -0.1 0.05 ± 47% perf-profile.self.cycles-pp.blk_mq_submit_bio
0.14 ± 20% -0.1 0.05 ± 86% perf-profile.self.cycles-pp.__io_req_task_work_add
0.14 ± 18% -0.1 0.05 ± 48% perf-profile.self.cycles-pp.__bio_release_pages
0.29 ± 13% -0.1 0.20 ± 20% perf-profile.self.cycles-pp.llist_reverse_order
0.17 ± 19% -0.1 0.09 ± 41% perf-profile.self.cycles-pp.memset_orig
0.13 ± 31% -0.1 0.06 ± 51% perf-profile.self.cycles-pp.xfs_inode_to_log_dinode
0.12 ± 14% -0.1 0.05 ± 47% perf-profile.self.cycles-pp.xfs_bmap_add_extent_unwritten_real
0.11 ± 23% -0.1 0.05 ± 73% perf-profile.self.cycles-pp.inode_maybe_inc_iversion
0.14 ± 12% -0.1 0.08 ± 14% perf-profile.self.cycles-pp.xfs_file_write_iter
0.14 ± 23% -0.1 0.09 ± 22% perf-profile.self.cycles-pp.fget
0.10 ± 13% -0.0 0.06 ± 46% perf-profile.self.cycles-pp.xfs_iext_get_extent
0.03 ±102% +0.1 0.09 ± 21% perf-profile.self.cycles-pp.pick_next_task_fair
0.02 ±141% +0.1 0.08 ± 38% perf-profile.self.cycles-pp.__enqueue_entity
0.10 ± 21% +0.1 0.16 ± 24% perf-profile.self.cycles-pp.irqentry_enter
0.01 ±223% +0.1 0.08 ± 37% perf-profile.self.cycles-pp.xas_load
0.00 +0.1 0.08 ± 18% perf-profile.self.cycles-pp.__blk_mq_free_request
0.01 ±223% +0.1 0.09 ± 22% perf-profile.self.cycles-pp.dd_dispatch_request
0.20 ± 21% +0.1 0.28 ± 16% perf-profile.self.cycles-pp.update_load_avg
0.00 +0.1 0.08 ± 25% perf-profile.self.cycles-pp.ttwu_do_activate
0.16 ± 31% +0.1 0.24 ± 20% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.08 ± 27% +0.1 0.17 ± 23% perf-profile.self.cycles-pp.__dd_dispatch_request
0.09 ± 21% +0.1 0.19 ± 25% perf-profile.self.cycles-pp.enqueue_task_fair
0.10 ± 14% +0.1 0.20 ± 23% perf-profile.self.cycles-pp.rb_erase
0.12 ± 32% +0.1 0.22 ± 12% perf-profile.self.cycles-pp.llist_add_batch
0.00 +0.1 0.11 ± 31% perf-profile.self.cycles-pp.prepare_to_wait_exclusive
0.54 ± 12% +0.1 0.67 ± 6% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.00 +0.1 0.14 ± 27% perf-profile.self.cycles-pp.sbitmap_queue_get_shallow
0.01 ±223% +0.1 0.15 ± 22% perf-profile.self.cycles-pp.xa_find_after
0.73 ± 11% +0.1 0.87 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.02 ±142% +0.2 0.17 ± 22% perf-profile.self.cycles-pp.dd_has_work
0.36 ± 13% +0.2 0.52 ± 12% perf-profile.self.cycles-pp.idle_cpu
0.10 ± 25% +0.2 0.26 ± 15% perf-profile.self.cycles-pp._find_next_zero_bit
0.86 ± 5% +0.2 1.02 ± 12% perf-profile.self.cycles-pp.menu_select
0.09 ± 27% +0.2 0.26 ± 14% perf-profile.self.cycles-pp.elv_rqhash_find
0.20 ± 12% +0.2 0.37 ± 14% perf-profile.self.cycles-pp._find_next_and_bit
0.08 ± 53% +0.2 0.26 ± 19% perf-profile.self.cycles-pp.complete_cmd_fusion
0.00 +0.2 0.19 ± 26% perf-profile.self.cycles-pp.__sbitmap_weight
0.24 ± 11% +0.2 0.48 ± 12% perf-profile.self.cycles-pp.__schedule
0.21 ± 7% +0.2 0.46 ± 11% perf-profile.self.cycles-pp.cpu_util
0.01 ±223% +0.3 0.28 ± 15% perf-profile.self.cycles-pp.scsi_mq_get_budget
0.03 ±101% +0.3 0.30 ± 21% perf-profile.self.cycles-pp.sbitmap_get
0.08 ± 24% +0.4 0.48 ± 5% perf-profile.self.cycles-pp.sbitmap_find_bit
1.37 ± 8% +1.2 2.58 ± 8% perf-profile.self.cycles-pp.update_sg_lb_stats
0.82 ± 12% +5.0 5.84 ± 3% perf-profile.self.cycles-pp.poll_idle
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
