Hello, kernel test robot noticed a 11.1% improvement of stress-ng.kvm.ops_per_sec on: commit: d96c77bd4eeba469bddbbb14323d2191684da82a ("KVM: x86: switch hugepage recovery thread to vhost_task") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master testcase: stress-ng config: x86_64-rhel-9.4 compiler: gcc-12 test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory parameters: nr_threads: 100% testtime: 60s test: kvm cpufreq_governor: performance Details are as below: --------------------------------------------------------------------------------------------------> The kernel config and materials to reproduce are available at: https://download.01.org/0day-ci/archive/20241209/202412092202.7aae9e89-lkp@xxxxxxxxx ========================================================================================= compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: gcc-12/performance/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp7/kvm/stress-ng/60s commit: 0586ade9e7 ("Merge tag 'loongarch-kvm-6.13' of git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson into HEAD") d96c77bd4e ("KVM: x86: switch hugepage recovery thread to vhost_task") 0586ade9e7f9491c d96c77bd4eeba469bddbbb14323 ---------------- --------------------------- %stddev %change %stddev \ | \ 723877 ± 16% +35.2% 978789 ± 10% numa-meminfo.node1.AnonHugePages 1442 ± 9% +17.5% 1695 ± 6% perf-c2c.HITM.local 2.528e+09 ± 2% -11.6% 2.235e+09 ± 2% cpuidle..time 3569356 ± 3% -21.0% 2819592 ± 4% cpuidle..usage 64.52 -10.9% 57.49 ± 2% vmstat.cpu.id 23.34 ± 4% +30.1% 30.36 ± 10% vmstat.procs.r 13939 -12.8% 12150 ± 2% vmstat.system.cs 0.79 ± 2% +0.1 0.89 ± 2% mpstat.cpu.all.guest% 63.44 ± 2% -7.2 56.20 ± 2% mpstat.cpu.all.idle% 0.48 ± 4% +0.1 0.54 ± 3% mpstat.cpu.all.irq% 32.33 ± 3% +7.0 39.30 ± 3% mpstat.cpu.all.sys% 443.20 ± 8% +19.5% 529.74 ± 10% numa-vmstat.node0.nr_anon_transparent_hugepages 315827 ± 6% +27.5% 402638 ± 6% numa-vmstat.node1.nr_active_anon 275814 ± 8% +31.5% 362699 ± 8% numa-vmstat.node1.nr_anon_pages 342.47 ± 11% +51.8% 519.83 ± 6% numa-vmstat.node1.nr_anon_transparent_hugepages 315773 ± 6% +27.5% 402610 ± 6% numa-vmstat.node1.nr_zone_active_anon 17472 ± 2% +11.1% 19402 ± 2% stress-ng.kvm.ops 290.78 ± 2% +11.1% 322.94 ± 2% stress-ng.kvm.ops_per_sec 51303 ± 7% +58.7% 81441 ± 11% stress-ng.time.involuntary_context_switches 2146 ± 3% +20.6% 2588 ± 3% stress-ng.time.percent_of_cpu_this_job_got 1257 ± 3% +20.8% 1519 ± 3% stress-ng.time.system_time 104856 ± 2% -6.6% 97892 ± 2% stress-ng.time.voluntary_context_switches 278787 ± 13% +126.1% 630300 ± 7% sched_debug.cfs_rq:/.avg_vruntime.min 227.71 ± 63% -71.0% 65.95 ± 45% sched_debug.cfs_rq:/.load_avg.avg 824.16 ± 78% -72.5% 226.40 ± 71% sched_debug.cfs_rq:/.load_avg.stddev 278787 ± 13% +126.1% 630300 ± 7% sched_debug.cfs_rq:/.min_vruntime.min 222108 ± 6% -11.3% 197050 ± 3% sched_debug.cpu.avg_idle.stddev 8649 ± 2% -11.5% 7656 ± 2% sched_debug.cpu.nr_switches.avg -54.00 -30.7% -37.42 sched_debug.cpu.nr_uninterruptible.min 15.65 ± 7% -40.8% 9.26 ± 14% sched_debug.cpu.nr_uninterruptible.stddev 611289 ± 10% +24.8% 763168 ± 6% proc-vmstat.nr_active_anon 558549 ± 11% +26.6% 707278 ± 6% proc-vmstat.nr_anon_pages 761.04 ± 16% +37.4% 1045 ± 8% proc-vmstat.nr_anon_transparent_hugepages 17892 -4.4% 17106 proc-vmstat.nr_kernel_stack 48984 ± 2% +5.9% 51878 ± 2% proc-vmstat.nr_shmem 43444 -4.0% 41719 proc-vmstat.nr_slab_unreclaimable 611282 ± 10% +24.8% 763164 ± 6% proc-vmstat.nr_zone_active_anon 5143 ± 9% -98.6% 73.17 ± 49% proc-vmstat.numa_huge_pte_updates 2764317 ± 7% -93.8% 172403 ± 14% proc-vmstat.numa_pte_updates 5.598e+08 ± 2% +11.7% 6.255e+08 proc-vmstat.pgalloc_normal 5.596e+08 ± 2% +11.7% 6.253e+08 proc-vmstat.pgfree 1071233 ± 2% +11.7% 1196235 ± 2% proc-vmstat.thp_fault_alloc 74.39 ± 2% +5.3% 78.31 ± 2% perf-stat.i.MPKI 1.724e+09 ± 2% +6.5% 1.836e+09 perf-stat.i.branch-instructions 2.13 ± 4% -0.2 1.92 ± 2% perf-stat.i.branch-miss-rate% 6.151e+08 ± 2% +12.2% 6.899e+08 ± 2% perf-stat.i.cache-misses 6.512e+08 ± 2% +11.9% 7.288e+08 ± 2% perf-stat.i.cache-references 14247 -13.6% 12308 ± 2% perf-stat.i.context-switches 8.84 ± 2% +11.7% 9.87 ± 4% perf-stat.i.cpi 7.365e+10 ± 3% +18.9% 8.755e+10 ± 3% perf-stat.i.cpu-cycles 8.511e+09 ± 2% +6.4% 9.056e+09 perf-stat.i.instructions 0.12 ± 4% -10.9% 0.11 ± 3% perf-stat.i.ipc 72.42 ± 2% +5.2% 76.22 ± 2% perf-stat.overall.MPKI 2.16 ± 4% -0.2 1.95 ± 2% perf-stat.overall.branch-miss-rate% 8.67 ± 3% +11.6% 9.67 ± 3% perf-stat.overall.cpi 119.66 +6.0% 126.87 perf-stat.overall.cycles-between-cache-misses 0.12 ± 3% -10.4% 0.10 ± 3% perf-stat.overall.ipc 1.693e+09 ± 2% +6.5% 1.804e+09 perf-stat.ps.branch-instructions 6.048e+08 ± 2% +12.1% 6.777e+08 ± 2% perf-stat.ps.cache-misses 6.404e+08 ± 2% +11.8% 7.161e+08 perf-stat.ps.cache-references 14002 -13.6% 12095 ± 2% perf-stat.ps.context-switches 7.239e+10 ± 3% +18.8% 8.602e+10 ± 3% perf-stat.ps.cpu-cycles 8.356e+09 ± 2% +6.4% 8.894e+09 perf-stat.ps.instructions 5.138e+11 ± 3% +6.1% 5.45e+11 perf-stat.total.instructions 2.21 ± 8% -0.5 1.66 ± 10% perf-profile.calltrace.cycles-pp.common_startup_64 2.15 ± 8% -0.5 1.61 ± 10% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64 2.15 ± 8% -0.5 1.61 ± 10% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64 2.14 ± 8% -0.5 1.61 ± 10% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64 2.04 ± 8% -0.5 1.52 ± 10% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64 1.93 ± 8% -0.5 1.45 ± 11% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 1.88 ± 8% -0.5 1.40 ± 11% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 0.65 ± 6% -0.3 0.35 ± 71% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 1.07 ± 9% -0.3 0.79 ± 11% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 2.21 ± 8% -0.5 1.66 ± 10% perf-profile.children.cycles-pp.common_startup_64 2.21 ± 8% -0.5 1.66 ± 10% perf-profile.children.cycles-pp.cpu_startup_entry 2.21 ± 8% -0.5 1.66 ± 10% perf-profile.children.cycles-pp.do_idle 2.15 ± 8% -0.5 1.61 ± 10% perf-profile.children.cycles-pp.start_secondary 2.09 ± 8% -0.5 1.58 ± 11% perf-profile.children.cycles-pp.cpuidle_idle_call 1.98 ± 8% -0.5 1.49 ± 11% perf-profile.children.cycles-pp.cpuidle_enter 1.98 ± 8% -0.5 1.49 ± 11% perf-profile.children.cycles-pp.cpuidle_enter_state 1.10 ± 9% -0.3 0.82 ± 11% perf-profile.children.cycles-pp.intel_idle 0.54 ± 68% -0.3 0.28 ± 7% perf-profile.children.cycles-pp.ret_from_fork 0.54 ± 68% -0.3 0.28 ± 7% perf-profile.children.cycles-pp.ret_from_fork_asm 0.48 ± 76% -0.2 0.24 ± 7% perf-profile.children.cycles-pp.kthread 1.85 ± 2% -0.1 1.73 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 1.57 ± 2% -0.1 1.45 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.34 ± 5% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.vmx_vcpu_run 0.22 ± 7% -0.0 0.18 ± 10% perf-profile.children.cycles-pp.ktime_get 0.16 ± 5% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.sched_balance_rq 0.24 ± 3% -0.0 0.20 perf-profile.children.cycles-pp.__schedule 0.18 ± 2% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.schedule 0.24 ± 6% -0.0 0.20 ± 7% perf-profile.children.cycles-pp.handle_softirqs 0.26 ± 5% -0.0 0.22 ± 7% perf-profile.children.cycles-pp.__irq_exit_rcu 0.08 ± 14% -0.0 0.04 ± 44% perf-profile.children.cycles-pp.mutex_lock_killable 0.12 ± 7% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.update_sd_lb_stats 0.12 ± 6% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.sched_balance_find_src_group 0.14 ± 5% -0.0 0.11 ± 6% perf-profile.children.cycles-pp.pick_next_task_fair 0.11 ± 8% -0.0 0.08 ± 4% perf-profile.children.cycles-pp.update_sg_lb_stats 0.14 ± 3% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.__pick_next_task 0.12 ± 10% -0.0 0.09 ± 14% perf-profile.children.cycles-pp.tick_irq_enter 0.12 ± 7% -0.0 0.09 perf-profile.children.cycles-pp.sched_balance_newidle 0.12 ± 6% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 0.10 ± 4% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.05 +0.0 0.06 perf-profile.children.cycles-pp.kvm_skip_emulated_instruction 0.22 ± 4% +0.0 0.25 ± 4% perf-profile.children.cycles-pp.__kmalloc_cache_noprof 0.10 ± 4% +0.0 0.15 ± 4% perf-profile.children.cycles-pp.vmx_handle_exit 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.kvm_mmu_post_init_vm 0.18 ± 10% +0.1 0.28 ± 13% perf-profile.children.cycles-pp.mntput_no_expire 0.35 ± 4% +0.1 0.45 ± 8% perf-profile.children.cycles-pp.kvm_destroy_vm_debugfs 0.33 ± 4% +0.1 0.43 ± 8% perf-profile.children.cycles-pp.simple_recursive_removal 0.33 ± 4% +0.1 0.44 ± 8% perf-profile.children.cycles-pp.debugfs_remove 0.22 ± 13% +0.1 0.34 ± 17% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 95.03 +0.7 95.78 perf-profile.children.cycles-pp.do_syscall_64 95.06 +0.7 95.80 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 1.10 ± 9% -0.3 0.82 ± 11% perf-profile.self.cycles-pp.intel_idle 0.21 ± 7% -0.0 0.16 ± 10% perf-profile.self.cycles-pp.ktime_get 0.07 ± 14% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.cpuidle_enter_state 0.09 ± 4% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.update_sg_lb_stats 0.05 ± 7% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.__kmalloc_cache_noprof 0.22 ± 13% +0.1 0.34 ± 17% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.03 ± 71% +274.8% 0.10 ± 60% perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.vma_alloc_folio_noprof 0.02 ±124% +1183.3% 0.22 ±134% perf-sched.sch_delay.avg.ms.__cond_resched.__kmalloc_cache_noprof.kvm_init_irq_routing.kvm_create_vm.kvm_dev_ioctl 0.01 ±104% +2913.6% 0.22 ±195% perf-sched.sch_delay.avg.ms.__cond_resched.__kmalloc_cache_noprof.kvm_vcpu_ioctl.__x64_sys_ioctl.do_syscall_64 0.09 ± 14% -33.2% 0.06 ± 25% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.46 ± 50% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm.kvm_create_vm 0.07 ± 55% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node 0.01 ± 85% +173.8% 0.02 ± 36% perf-sched.sch_delay.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork 0.07 ±117% -93.8% 0.00 ±167% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate.isra 0.01 ± 22% -100.0% 0.00 perf-sched.sch_delay.avg.ms.kthreadd.ret_from_fork.ret_from_fork_asm 0.01 ± 23% -100.0% 0.00 perf-sched.sch_delay.avg.ms.percpu_down_write.cgroup_attach_task_all.kvm_vm_worker_thread.kthread 0.02 ± 54% -100.0% 0.00 perf-sched.sch_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.cgroup_css_set_fork.cgroup_can_fork 0.02 ± 37% -100.0% 0.00 perf-sched.sch_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.exit_signals.do_exit 0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__kthread_parkme.kvm_vm_worker_thread.kthread 0.00 ± 8% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.kthread.ret_from_fork.ret_from_fork_asm 0.02 ± 26% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.kthread_stop.kvm_destroy_vm 0.03 ± 44% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm 0.22 ± 62% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 0.01 ± 52% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.kvm_nx_huge_page_recovery_worker.kvm_vm_worker_thread.kthread 0.03 ±137% +872.7% 0.25 ±120% perf-sched.sch_delay.max.ms.__cond_resched.__kmalloc_cache_noprof.kvm_init_irq_routing.kvm_create_vm.kvm_dev_ioctl 0.01 ±104% +6604.2% 0.54 ±205% perf-sched.sch_delay.max.ms.__cond_resched.__kmalloc_cache_noprof.kvm_vcpu_ioctl.__x64_sys_ioctl.do_syscall_64 2.13 ± 64% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm.kvm_create_vm 0.99 ± 92% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node 0.49 ±179% -99.1% 0.00 ±167% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate.isra 0.56 ± 54% +305.5% 2.28 ±125% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 0.92 ± 56% -100.0% 0.00 perf-sched.sch_delay.max.ms.kthreadd.ret_from_fork.ret_from_fork_asm 0.02 ± 42% -100.0% 0.00 perf-sched.sch_delay.max.ms.percpu_down_write.cgroup_attach_task_all.kvm_vm_worker_thread.kthread 0.41 ± 73% -100.0% 0.00 perf-sched.sch_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.cgroup_css_set_fork.cgroup_can_fork 0.04 ± 71% -100.0% 0.00 perf-sched.sch_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.exit_signals.do_exit 0.59 ± 65% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_preempt_disabled.__kthread_parkme.kvm_vm_worker_thread.kthread 0.14 ± 59% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_preempt_disabled.kthread.ret_from_fork.ret_from_fork_asm 4.18 ± 15% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.kthread_stop.kvm_destroy_vm 3.41 ± 52% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm 102.46 ± 45% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 1.29 ± 59% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.kvm_nx_huge_page_recovery_worker.kvm_vm_worker_thread.kthread 0.09 ± 32% +49267.1% 46.08 ± 28% perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.__synchronize_srcu.part.0 109.95 ± 23% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node 126.03 ± 17% -51.4% 61.29 ± 82% perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.mm_take_all_locks.__mmu_notifier_register.mmu_notifier_register 0.47 ± 84% +4864.0% 23.41 ± 56% perf-sched.wait_and_delay.avg.ms.__cond_resched.folio_zero_user.__do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 3.11 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.kthread_exit.kthread.ret_from_fork 3.73 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.kthreadd.ret_from_fork.ret_from_fork_asm 69.11 ± 15% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.percpu_rwsem_wait.__percpu_down_read.exit_signals.do_exit 0.05 ± 36% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.__kthread_parkme.kvm_vm_worker_thread.kthread 0.00 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.kthread.ret_from_fork.ret_from_fork_asm 0.05 ± 44% +649.5% 0.34 ± 43% perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.__synchronize_srcu.part.0 0.05 ± 25% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.kthread_stop.kvm_destroy_vm 0.12 ± 21% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm 115.49 ± 5% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 0.05 ± 54% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kvm_nx_huge_page_recovery_worker.kvm_vm_worker_thread.kthread 443.01 ± 3% +16.6% 516.61 ± 9% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1956 ± 2% +15.5% 2260 ± 7% perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.__synchronize_srcu.part.0 29.33 ± 87% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node 1533 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.kthread_exit.kthread.ret_from_fork 1295 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.kthreadd.ret_from_fork.ret_from_fork_asm 8.67 ± 40% -100.0% 0.00 perf-sched.wait_and_delay.count.percpu_rwsem_wait.__percpu_down_read.exit_signals.do_exit 1532 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_preempt_disabled.__kthread_parkme.kvm_vm_worker_thread.kthread 1533 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_preempt_disabled.kthread.ret_from_fork.ret_from_fork_asm 1527 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.kthread_stop.kvm_destroy_vm 1512 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm 1520 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 1533 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.kvm_nx_huge_page_recovery_worker.kvm_vm_worker_thread.kthread 1206 ± 2% -22.7% 932.83 ± 7% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 9.06 ± 58% +2759.4% 259.12 ± 3% perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.__synchronize_srcu.part.0 192.18 ± 22% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node 19.46 ± 81% +1187.5% 250.50 ± 9% perf-sched.wait_and_delay.max.ms.__cond_resched.folio_zero_user.__do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 170.49 ± 13% -41.8% 99.23 ± 58% perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_lru_noprof.alloc_inode.new_inode.__debugfs_create_file 85.98 ± 16% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.kthread_exit.kthread.ret_from_fork 166.34 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.kthreadd.ret_from_fork.ret_from_fork_asm 135.27 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.percpu_rwsem_wait.__percpu_down_read.exit_signals.do_exit 3.82 ± 43% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__kthread_parkme.kvm_vm_worker_thread.kthread 250.93 ± 10% -18.2% 205.36 ± 15% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.mm_take_all_locks 0.14 ± 59% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.kthread.ret_from_fork.ret_from_fork_asm 28.03 ±172% +486.3% 164.36 ± 25% perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.__synchronize_srcu.part.0 6.38 ± 25% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.kthread_stop.kvm_destroy_vm 6.97 ± 48% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm 284.62 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 11.65 ±146% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kvm_nx_huge_page_recovery_worker.kvm_vm_worker_thread.kthread 0.03 ± 59% +43825.1% 14.86 ± 59% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.vma_alloc_folio_noprof 0.15 ± 44% +17885.5% 27.85 ± 79% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pte_alloc_one.__do_huge_pmd_anonymous_page 0.08 ± 67% +35500.4% 28.54 ± 68% perf-sched.wait_time.avg.ms.__cond_resched.__get_user_pages.populate_vma_page_range.__mm_populate.vm_mmap_pgoff 0.01 ± 86% +2505.9% 0.22 ±194% perf-sched.wait_time.avg.ms.__cond_resched.__kmalloc_cache_noprof.kvm_vcpu_ioctl.__x64_sys_ioctl.do_syscall_64 0.05 ± 30% +86818.3% 45.92 ± 28% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.__synchronize_srcu.part.0 0.57 ± 46% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm.kvm_create_vm 109.88 ± 23% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node 126.01 ± 17% -43.6% 71.10 ± 55% perf-sched.wait_time.avg.ms.__cond_resched.down_write.mm_take_all_locks.__mmu_notifier_register.mmu_notifier_register 0.24 ± 84% +9652.8% 23.26 ± 56% perf-sched.wait_time.avg.ms.__cond_resched.folio_zero_user.__do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 0.01 ± 72% +425.4% 0.06 ± 55% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.__kvm_mmu_topup_memory_cache.kvm_mmu_load.vcpu_enter_guest 0.17 ± 91% +54596.8% 92.80 ± 29% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.kvm_create_vm.kvm_dev_ioctl.__x64_sys_ioctl 0.03 ±139% +1.2e+05% 41.90 ±222% perf-sched.wait_time.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range 3.11 ± 2% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.kthread_exit.kthread.ret_from_fork 3.72 ± 3% -100.0% 0.00 perf-sched.wait_time.avg.ms.kthreadd.ret_from_fork.ret_from_fork_asm 8.21 ± 23% -100.0% 0.00 perf-sched.wait_time.avg.ms.percpu_down_write.cgroup_attach_task_all.kvm_vm_worker_thread.kthread 0.59 ±191% -100.0% 0.00 perf-sched.wait_time.avg.ms.percpu_rwsem_wait.__percpu_down_read.cgroup_css_set_fork.cgroup_can_fork 69.10 ± 15% -100.0% 0.00 perf-sched.wait_time.avg.ms.percpu_rwsem_wait.__percpu_down_read.exit_signals.do_exit 0.04 ± 42% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__kthread_parkme.kvm_vm_worker_thread.kthread 0.03 ± 66% +998.3% 0.32 ± 46% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.__synchronize_srcu.part.0 0.03 ± 29% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.kthread_stop.kvm_destroy_vm 0.09 ± 15% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm 115.27 ± 5% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 0.04 ± 54% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.kvm_nx_huge_page_recovery_worker.kvm_vm_worker_thread.kthread 442.99 ± 3% +16.6% 516.60 ± 9% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.38 ±195% +1156.7% 17.32 ± 39% perf-sched.wait_time.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.15 ±101% +93541.9% 142.02 ± 47% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.vma_alloc_folio_noprof 3.01 ± 49% +6033.5% 184.87 ± 13% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pte_alloc_one.__do_huge_pmd_anonymous_page 1.67 ±101% +13142.2% 220.53 ± 16% perf-sched.wait_time.max.ms.__cond_resched.__get_user_pages.populate_vma_page_range.__mm_populate.vm_mmap_pgoff 0.01 ± 87% +5859.3% 0.54 ±205% perf-sched.wait_time.max.ms.__cond_resched.__kmalloc_cache_noprof.kvm_vcpu_ioctl.__x64_sys_ioctl.do_syscall_64 0.04 ± 51% +2258.2% 1.01 ±126% perf-sched.wait_time.max.ms.__cond_resched.__tlb_batch_free_encoded_pages.tlb_finish_mmu.vms_clear_ptes.part 6.20 ±101% +4081.5% 259.10 ± 3% perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.__synchronize_srcu.part.0 2.36 ± 57% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm.kvm_create_vm 192.15 ± 22% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node 9.78 ± 80% +2460.3% 250.48 ± 9% perf-sched.wait_time.max.ms.__cond_resched.folio_zero_user.__do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault 170.47 ± 13% -31.8% 116.31 ± 32% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_lru_noprof.alloc_inode.new_inode.__debugfs_create_file 0.01 ± 72% +734.3% 0.09 ± 89% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.__kvm_mmu_topup_memory_cache.kvm_mmu_load.vcpu_enter_guest 1.87 ± 85% +8622.8% 162.93 ± 19% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.kvm_create_vm.kvm_dev_ioctl.__x64_sys_ioctl 0.16 ±188% +1.1e+05% 168.55 ±220% perf-sched.wait_time.max.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range 85.98 ± 16% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.kthread_exit.kthread.ret_from_fork 166.33 ± 3% -100.0% 0.00 perf-sched.wait_time.max.ms.kthreadd.ret_from_fork.ret_from_fork_asm 101.24 ± 10% -100.0% 0.00 perf-sched.wait_time.max.ms.percpu_down_write.cgroup_attach_task_all.kvm_vm_worker_thread.kthread 16.51 ±212% -100.0% 0.00 perf-sched.wait_time.max.ms.percpu_rwsem_wait.__percpu_down_read.cgroup_css_set_fork.cgroup_can_fork 135.25 ± 8% -100.0% 0.00 perf-sched.wait_time.max.ms.percpu_rwsem_wait.__percpu_down_read.exit_signals.do_exit 3.64 ± 43% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_preempt_disabled.__kthread_parkme.kvm_vm_worker_thread.kthread 250.91 ± 10% -18.2% 205.34 ± 15% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.mm_take_all_locks 25.20 ±193% +552.2% 164.34 ± 25% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__synchronize_srcu.part.0 3.37 ± 19% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.kthread_stop.kvm_destroy_vm 3.71 ± 42% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.kvm_vm_create_worker_thread.kvm_mmu_post_init_vm 284.60 ± 8% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 11.46 ±149% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.kvm_nx_huge_page_recovery_worker.kvm_vm_worker_thread.kthread Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki